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SmuUin  shows  that  getting  integrated  into  the  DoD  enterprise  and 
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discusses  the  benefits  of  hybrid  process  improvement  models. 
hj  Capers  Jones 


Is  CMMI  Useful  and  Usable  in  Small  Settings?  One 
Example 

This  article  presents  the  motivation,  the  processes  used,  and  the  major 
results  of  the  CMMI  for  a  Small  Business  pilot  from  the  perspective  of  the 
team  that  worked  on  the  pilot. 

bjj  Sandra  Cepeda,  Suzanne  Garcia,  and  Jacquelyn  Fanghout 


Why  Do  I  Need  All  That  Process?  Tm  Only  a  Small  Project 

This  article  shows  how  to  fix  the  issue  of  having  a  variety  of  project  sizes 
but  a  single  set  of  processes  originally  built  for  larger  projects. 
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Small  Project  Survival  Among  the  CMMI  Level  5 
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Good  Things  Come  in  Small  Packages 


C  C  ood  things  come  in  small  packages  ”  Most  people  have  probably  heard  this  say- 
Vj’ing  from  the  time  they  were  kids.  Small  things  just  don’t  seem  to  get  the  respect 
they  deserve  —  including  small  projects.  Most  of  the  articles  in  this  month’s 
Crosstalk  provide  insightful  advice  for  adapting  large  project  processes  for  small 
projects.  But  what  about  the  other  side  of  the  story?  Small  projects  have  much  to  offer. 
Maybe  larger  projects  should  consider  what  they  can  adapt  from  small  projects.  Three 
areas  that  large  projects  may  benefit  from  the  lessons  learned  by  smaller  projects  are 
Agile  techniques,  bottom-up  estimation,  and  communication  improvements. 

There  have  been  efforts  to  scale  Agile  practices  to  large  projects.  One  such  effort  involves 
breaking  large  projects  into  smaller  projects.  This  might  work  for  the  small  pieces,  but  what 
about  when  the  pieces  need  to  be  brought  back  together?  A  consistent  approach  is  still  needed 
to  pull  them  together.  However,  portions  of  Agile  programming  can  still  work  on  large  projects. 
Pair  programming  is  one  such  idea:  It  may  seem  reasonable  that  pairing  software  developers  has 
been  shown  to  decrease  errors,  and  contrary  to  conventional  logic,  pairing  up  programmers  also 
has  been  shown  to  increase  productivity.  The  cyclical  development  of  agile  programming  also 
scales  to  large  projects  and  is  discussed  in  depth  with  the  variations  of  the  Spiral  Model  which 
is  now  espoused  in  the  software  community.  Barry  Boehm  and  his  co-authors  discuss  this  thor¬ 
oughly  in  back  issues  of  CrOSSTalk. 

Bottom-up  software  estimation  is  another  technique  used  in  many  small  projects  that  may 
apply  to  large  projects.  While  it  usually  is  not  practical  to  use  bottom-up  estimates  at  the  begin¬ 
ning  of  a  large  project,  bottom-up  software  estimation  is  certainly  applicable  as  a  sanity  check 
of  the  estimates  as  the  project  progresses,  as  work  is  divided  into  smaller  pieces,  and  as  it  is 
assigned  to  smaller  groups  and  individuals. 

And,  of  course,  there  is  communication.  A  2002  CrossTalk  article  discusses  new  engineers 
in  large  companies  sitting  in  their  cubicles  and  accomplishing  little  because  they  don’t  want  to 
make  a  bother  of  themselves  by  asking  a  lot  of  questions.  Communication  tends  to  be  easier  on 
small  projects,  and  someone  just  sitting  will  certainly  be  noticed  more  readily.  Providing  tools 
for  communication  such  as  white  boards  and  open  space  can  be  implemented  in  large  compa¬ 
nies,  and  the  resulting  camaraderie  has  also  been  shown  to  improve  productivity. 

We  start  this  month’s  issue  with  Fred  SmuUin’s  story  of  one  small  project  finding  its  way  in 
the  big  Department  of  Defense  world  in  Navigating  The  Enterprise  Forest.  In  Development  Practices 
for  Small  Software  Applications^  Capers  Jones  shares  his  insights,  comparing  the  benefits  and  draw¬ 
backs  of  the  Capability  Maturity  Model  Integration  (CMMI)  with  Agile  development  for  small 
projects.  We  next  provide  a  series  of  articles  with  expertise  on  adapting  large  processes  for  small 
projects  including  Is  CMMI  Fseful  and  F sable  in  Small  Settings?  One  Example  by  Sandra  Cepeda, 
Suzanne  Garcia,  and  Jacquelyn  Langhout;  Whj  Do  I  Need  All  That  Process?  Em  Only  A  Small  Project 
by  Mark  Brodnik,  Robyn  Pious e,  and  Terry  Leip;  Small  Project  Survival  Among  the  CMMI  Eevel  5 
Big  Processes  by  Alan  C.  Jost;  and  Field  Guide  to  Provide  Step-by-Step  Examples  for  Improving  Processes 
in  Small  Settings  by  Caroline  Graettinger,  Suzanne  Garcia,  Christian  Carmody,  M.  Lynn  Penn,  and 
WiUiam  Peterson. 

On  a  side  note,  CrossTalk  will  be  celebrating  the  20'’"  anniversary  of  our  first  issue  in 
August.  We  would  like  to  celebrate  this  milestone  with  stories  of  how  CrossTalk  has  helped 
our  readers  save  time  and  money,  improve  processes,  salvage  projects,  and  make  your  jobs  eas¬ 
ier.  We  received  several  success  stories  with  our  survey  in  2004  -  those  responses  resulted  in  the 
continuation  of  this  journal.  Your  responses  now  will  enable  us  to  thank  our  co-sponsors  for 
their  continued  support  and  strengthen  our  position  as  we  seek  additional  co-sponsors. 

Large  projects  tend  to  fail  more  frequently  than  small  projects  for  a  variety  of  reasons.  While 
small  projects  are  figuring  out  how  to  benefit  from  the  practices  of  large  projects,  large  projects 
can  also  learn  lessons  from  small  projects.  Meanwhile,  CrossTalk  will  continue  to  offer 
strategies  for  both. 


Elizabeth  Starrett 
Publisher 


Small  Projects,  Big  Issues 


Navigating  the  Enterprise  Forest 


Fred  SmuUin 
Integratable  Technologies,  ITTC 

What  happens  when  you  take  a  local  depot  solution  and  promote  it for  worldwide  use?  You  suddenly  find  yourself  facing  the 
challenge  of  integrating  into  the  Department  of  Defense  (DoD)  enterprise  with  few  maps  to  guide  you.  I  will  share  my  lessons 
learned  as  the  former  chief  software  architect  of  the  G200  Defense  Kepair  Information  Togistics  System  (DRITS )  that 
started  as  a  local  solution  and  matured  into  an  Air  Force  solution  used  both  in  the  .com  and  .mil  domains. 


DRILS  began  as  a  local  maintenance 
data  collection  (MDC)  system  in 
2000  to  capture  the  serialized  repair  of 
assets.  It  was  commissioned  by  F-16  sup¬ 
ply  chain  managers  (SCMs)  to  connect 
them  in  real  time  to  depot  avionics  repair 
activities  at  Hill  Air  Force  Base.  The 
objective  was  to  coUect  and  analyze  vital 
repair  shop  information  in  order  to 
increase  the  reliability  and  availability  of 
F-16  avionics  as  well  as  decrease  repair 
costs  [1].  DRILS  most  importantly  facili¬ 
tated  documentation  of  individual  chips, 
resistors,  and  other  small  parts  being 
replaced  within  the  aircraft  avionics  com¬ 
ponents.  Other  Air  Force  MDC  systems 
were  not  able  to  provide  this  level  of 
detail,  which  turned  out  to  be  the  most 
important  data  to  SCMs  as  these  parts 
were  the  ones  that  actually  failed  within 
the  avionics  components.  The  informa¬ 
tion  analyzed  from  DRILS  by  the  F-16 
SCMs  under  their  Falcon  Flex  program  [2] 
has  enabled  $133  million  in  cost  avoidance 
since  2000  and  has  been  projected  to 
achieve  approximately  $678  million 
through  the  aircraft  end  of  life  [3]. 

I  will  admit  that  the  DRILS  stakehold¬ 
ers  were  naive  to  the  DoD  approval 
requirements  for  an  Information 
Technology  (IT)  system.  It  would  take  us 
more  than  a  year  after  fielding  our  initial 
prototype  to  navigate  our  way  and  be  rec¬ 
ognized  as  a  legitimate  IT  system. 

The  DRILS  story  is  not  unlike  those 
of  other  locally  grown  data  systems  in  the 
DoD.  An  information  gap  existed  within 
the  current  mix  of  maintenance  informa¬ 
tion  systems  and  the  depot  organization 
took  steps  to  plug  that  gap  which  eventu¬ 
ally  gave  birth  to  DRILS.  You  do  not  have 
to  look  far  to  find  information  gaps  in  the 
DoD.  We  live  in  a  data  rich  environment, 
yet  we  are  information  poor;  someone 
somewhere  does  not  have  access  to  the 
information  they  need.  This  basic  hunger 
for  dependable  information,  as  well  as  the 
lead  time  and  funding  required  to  modify 
legacy  systems,  leads  to  creation  of  local 


stovepipe  solutions  funded  and  built  by 
the  user  in  that  domain. 

Tight  budgets  are  impacting  the  ability 
to  implement  local  solutions.  A  2004  U.S. 
General  Accounting  Office  (GAO)  report 
found  that  the  DoD  requested  approxi¬ 
mately  $19  billion  for  fiscal  year  2004  to 
operate,  maintain,  and  modernize  2,274 
business  systems.  The  report  also  identi¬ 
fied  that  uncontrolled  DoD  spending 
resulted  in  stovepiped  and  duplicative  sys¬ 
tems  that  included  more  than  200  inven¬ 
tory  and  450  personnel  systems  being  pro¬ 
cured  and  sustained.  Very  often  these 
stovepipe  solutions  get  thrown  over  the 
wall  to  DoD  IT  organizations  to  integrate 
and  sustain  within  the  enterprise  [4].  The 
costs  of  the  integration  and  sustainment 
efforts  result  in  priorities  getting  shifted 
within  existing  budgets  to  accommodate 
these  unplanned  requirements. 

The  National  Defense  Authorization 
Act  (NDAA)  of  2005  was  crafted  to 
specifically  address  this  issue  [5].  It 
requires  the  DoD  comptroller  to  deter¬ 
mine  that  system  improvements  exceeding 
$1  million  meet  the  criteria  specified  in  the 
act.  DoD  portfolio  management  (PfM) 
initiatives  have  been  introduced  to  comply 
with  the  NDAA  requirement  as  well  as 
limit  the  flow  of  local  solutions  that  may 
be  stovepiped  or  duplicative.  However, 
the  door  is  not  completely  closed  to 
approval  of  local  systems.  You  can  get 
your  local  solution  approved  to  operate  if 
you  know  how  to  navigate  your  way 
through  the  forest.  Based  on  my  experi¬ 
ence,  if  you  take  the  time  to  address  the 
following  questions,  you  wiU  increase  your 
likelihood  of  surviving  in  the  enterprise. 

What  Gaps  Are  You  Filling? 

Identify  what  gaps  you  are  filling  in  the 
information  food  chain.  Is  there  a  reason 
why  no  one  else  is  providing  the  informa¬ 
tion?  This  is  your  critical  foundation  upon 
which  everything  else  is  built.  If  you  are 
merely  churning  out  the  same  information 
that  other  systems  are  producing,  you  will 


not  get  far.  Focus  on  those  information 
gaps  that  prompted  you  to  build  the  sys¬ 
tem. 

Take  those  gaps  and  then  describe 
what  is  in  it  for  the  user  community,  espe¬ 
cially  the  person  doing  data  entry.  Are  they 
getting  more  out  of  it  than  what  they  put 
in?  If  it  is  not  useful  to  them,  you  are  not 
going  to  get  your  dependable  data.  Make  it 
worthwhile  for  them,  and  you  will  never 
be  short  of  dependable  data. 

Any  requirement,  given  enough  time 
and  money,  can  be  integrated  into  any 
legacy  system.  One  of  the  most  common 
reasons  a  requirement  is  not  integrated  is 
that  few  organizations  have  the  time 
and/or  money  legacy  systems  request  to 
implement  a  new  requirement.  The  user 
community  finds  it  cheaper  and  faster  in 
the  near  term  to  implement  their  require¬ 
ments  to  fill  the  information  gap,  and 
hope  to  interface  it  with  a  legacy  system 
down  the  road.  Unfortunately,  this  is 
counterproductive  as  it  just  creates  more 
stovepipes. 

Try  to  identify  if  legacy  systems  have 
plans  to  plug  this  information  gap,  and  if 
so,  when.  If  there  are  plans  but  they  are 
years  out  on  the  horizon,  you  may  get 
interim  authority  to  operate  your  gap-fiU- 
ing  solution  that  could  evolve  into  a  long¬ 
term  solution  if  the  implementation  is 
done  well. 

Develop  your  own  comparison  matrix 
of  legacy  systems  that  perform  similar 
functions  or  may  potentially  interface  to 
your  system.  Try  to  be  impartial  in  your 
evaluation  in  order  to  maximize  its  credi¬ 
bility.  Contact  the  legacy  systems  and  edu¬ 
cate  them  about  what  you  are  doing. 
Approach  them  with  a  partnership  offer 
that  assists  them  with  improving  the  qual¬ 
ity  of  data  and  information  in  their  sys¬ 
tem.  Document  which  systems  you  would 
interface  with  if  you  could  and  why. 
Estimate  interface  costs  and  return  on 
investment  where  possible.  Interfaces  to 
legacy  systems  usually  provide  returns  on 
investment  in  the  areas  of  duplicate  data 
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entry  reduction,  minimization  of  data 
entry  errors,  improved  data  dependability, 
and  near  real-time  data  updates  across  the 
enterprise.  Keep  in  mind  that  there  are 
two  costs  to  any  user  interface:  yours  and 
the  legacy  system.  Your  user  community 
may  have  to  pay  for  both. 

In  the  case  of  DRILS,  we  saw  that  it 
could  potentially  interface  with  the 
Reliability  and  Maintainability  Infor¬ 
mation  System  (REMIS)  and  the  Core 
Automated  Maintenance  System  (CAMS). 
Users  were  frustrated  with  data  entry 
processes  and  business  rules  they  per¬ 
ceived  to  be  cumbersome  as  well  as  with 
challenges  they  encountered  trying  to  ana¬ 
lyze  historical  data.  We  focused  our  efforts 
on  streamlining  the  data  entry  and  data 
analysis  processes  for  our  users.  On  the 
depot  shop  floor,  we  were  able  to  cut  data 
entry  time  by  80  percent  on  average  com¬ 
pared  to  the  legacy  system.  The  result  was 
that  the  volume  of  depot  maintenance 
data  actually  increased  from  the  shop 
floor  when  compared  to  the  legacy  data 
system.  Interfacing  with  the  legacy  sys¬ 
tems  turned  out  to  be  the  greatest 
approval  challenge.  It  took  nearly  six  years 
of  meetings,  briefings,  and  requests  for 
funding  before  the  official  system  inter¬ 
face  was  allowed  to  be  built  with  REMIS. 
A  CAMS  interface  is  still  actively  being 
pursued  at  this  date. 

One  lesson  learned  is  that  headquar¬ 
ters  is  generally  willing  to  entertain  tem¬ 
porary  solutions  to  initially  fiU  informa¬ 
tion  gaps.  Locally  developed  temporary 
solutions  are  usually  more  agile  and  less 
costly  than  legacy  systems  to  experiment 
with.  Thus,  temporary  solutions  are  excel¬ 
lent  proving  grounds  for  defining  require¬ 
ments  to  be  incorporated  into  the  legacy 
system  in  the  long  term. 

This  proved  true  for  DRILS:  Our  user 
community  encompassed  a  relatively  large 
portion  of  the  F-16  avionics  community 
but  was  still  small  when  compared  to  Air 
Force-wide  MDC  legacy  systems.  Our 
development  and  support  teams  were  also 
much  smaller  which  shortened  our  deci¬ 
sion-making  time.  The  architecture  design 
allowed  us  to  isolate  experimental  mod¬ 
ules  from  all  user  communities  except 
those  participating.  Thus,  our  cost  to 
implement  requested  changes  for  experi¬ 
mental  initiatives  was  significantly  smaller 
and  our  time  to  value  was  also  significant¬ 
ly  shorter.  This  made  DRILS  an  ideal  sys¬ 
tem  to  support  MDC  experiments  such  as 
Air  Force  Serial  Number  Tracking  initia¬ 
tives  sponsored  by  high-level  champions. 

Who  Are  Your  Champions? 

A  key  to  successfully  implementing  any  IT 


system  in  the  DoD  is  to  identify  champi¬ 
ons  within  the  customer  and  user  base. 
Champions  identified  within  this  commu¬ 
nity  can  help  advocate  the  system  at  the 
various  levels  of  review  and  approval. 
These  champions  need  to  be  evangelistic 
because  their  support  wiU  be  tested  up  the 
chain  of  approval.  They  wiU  need  to  be 
able  to  communicate  their  need  and  why 
your  solution  is  the  best. 

For  example,  in  the  case  of  DRILS,  we 
were  fortunate  that  the  product  concept 
and  implementation  sold  itself  Several  lev¬ 
els  of  champions  sprung  up  during  the 
product  implementation.  The  SCMs  want¬ 
ed  the  repair  data  from  the  shop  floor  and 
convinced  the  repair  shop  supervision  to 
give  it  a  try.  Supervision  asked  their  data 
entry  technicians  to  try  the  system.  They 
did  with  some  reluctance,  but  once  they 
started  entering  data,  they  became  believers. 

key  to  successfully 
implementing  any  IT 
system  in  the  DoD  is  to 
identify  champions  within 
the  customer  and  user 
base.  Champions 
identified  within  this 
community  can  help 
advocate  the  system  at 
the  various  levels  of 
review  and  approval.^* 

Technicians  in  depot  repair  shops  are 
graded  on  their  production,  and  the  over¬ 
all  organization  is  graded  on  its  ability  to 
produce  quality  assets  on  schedule  and 
on  budget.  Any  impediment  to  those 
goals  can  impact  their  customers  and 
cost  them  workload  in  their  competitive 
environment.  Legacy  MDC  systems  used 
until  then  had  been  deemed  cumbersome 
to  use  by  those  using  it  and  did  not  pro¬ 
vide  any  perceivable  value  to  those  tech¬ 
nicians  in  meeting  quality,  budget,  and 
schedule.  The  DRILS  development  team 
lived  on  the  shop  floor  for  nearly  four 
years  working  hand  in  hand  with  the 
using  technicians  to  refine  how  the  data 
was  collected  and  displayed.  This  enabled 
the  system  to  provide  immediate  payback 
to  the  person  entering  the  data. 


This  focus  on  the  technician  at  the 
point  of  maintenance  enabled  them  to 
proactively  identify  issues  that  most  like¬ 
ly  would  not  have  been  detected  with  the 
legacy  systems.  A  real-world  example 
involved  the  F-16  multi- function  displays 
whose  newly  manufactured  replacement 
power  supplies  that  cost  $5,000  each 
were  failing  within  five  months  of  instal¬ 
lation.  The  DRILS  design  enabled  the 
technician  to  easily  notice  the  trend,  stop 
installing  those  parts,  and  alert  the  SCM 
who  triggered  an  investigation  with  the 
manufacturer.  That  investigation  eventu¬ 
ally  led  to  the  identification  of  a  defect  in 
the  manufacturing  process.  Without 
DRILS,  the  trend  may  have  gone  on  for 
many  more  months  and  possibly  ground¬ 
ed  aircraft  due  to  failed  parts  clogging 
the  supply  chain  and  consuming  financial 
resources. 

Stories  such  as  these  enabled  long¬ 
standing  issues  to  start  getting  fixed. 
These  success  stories  gained  the  attention 
of  the  warfighter  customer  who  then 
wanted  the  system  adapted  for  their  use. 
This  sold  the  supervisor  who  in  turn  sold 
their  Colonel  who  in  turn  sold  his 
Brigadier  General.  The  Brigadier  General 
then  raised  awareness  aU  the  way  to  the 
Office  of  the  Secretary  of  Defense.  Word 
started  to  spread  between  the  weapon  sys¬ 
tems  and  Major  Commands  when 
warfighters  who  had  used  the  system 
moved  from  unit  to  unit.  It  was  not  long 
before  we  had  obtained  several  levels  of 
champions.  But  our  most  critical  level 
continued  to  be  the  data  entry  person  at 
the  point  of  maintenance. 

How  Do  You  Align  With  the 
Mission? 

Another  key  to  winning  acceptance  in 
the  DoD  enterprise  is  to  identify  how 
you  align  with  the  overall  IT  mission  of 
your  agency  and  the  DoD  in  general. 
Obtain  copies  of  headquarters  briefings 
in  your  domain  and  examine  their  road 
map  and  the  issues  they  are  trying  to 
solve.  How  do  you  fit  within  that  road 
map?  If  you  can  show  how  you  fit  with¬ 
in  that  road  map,  you  can  gain  critical 
awareness  and  possibly  acceptance  at  the 
headquarters  level.  Part  of  gaining  accep¬ 
tance  is  educating  them  about  how  you 
fit  with  current  legacy  systems. 

Compliance  With  Standards 

The  IT  industry  continues  to  evolve 
toward  a  net-centric  world  where  stan- 
dards-based  computing  is  pushing  out 
proprietary  products  in  order  to  facilitate 
easier  integration  in  heterogeneous  envi- 
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ronments.  The  DoD  continues  to  gravi¬ 
tate  to  these  standards,  although  slower 
than  industry,  for  the  same  reasons.  It  is 
important  that  you  inventory  applicable 
technology  standards  in  your  application 
domain  as  well  as  those  of  the  mission 
you  are  supporting  and  remain  consistent 
with  those  established  standards. 

DRILS  is  a  Web-based  Air  Force 
maintenance  data  collection  system;  well- 
published  standards  for  maintenance 
data  existed  in  Technical  Order  00-20-2 
and  other  publications  [6].  We  had  to 
remain  consistent  with  those  standards  at 
a  minimum  in  order  to  be  able  to  feed 
data  to  the  legacy  MDC  systems  in  order 
to  allow  a  much  broader  Air  Force  audi¬ 
ence  to  analyze  the  data.  Technology- 
wise,  we  intentionally  selected  well- 
known  commercial  off-the-shelf  prod¬ 
ucts  and  kept  those  products  to  a  mini¬ 
mum  in  order  to  avoid  integration 
headaches  while  remaining  consistent 
with  the  Global  Combat  Support  System 
—  Air  Force  requirements. 

Can  You  Participate  in  a 
Pathfinder  Initiative? 

Pathfinder  initiatives  are  very  beneficial. 
Merriam  Webster’s  online  dictionary 
defines  pathfinder  as,  ‘‘one  that  discovers 
a  way;  especiallj.  one  that  explores  un tra¬ 
versed  regions  to  mark  out  a  new  route” 
[7].  My  experience  with  pathfinder  initia¬ 
tives  has  involved  a  charter  between  a  par¬ 
ticular  community  of  interest  and  the 
headquarters  to  solve  a  process  or  infor¬ 
mation  gap.  These  pathfinders  involve 
assembling  members  of  the  community  of 
interest  to  review  and  improve  processes 
and  policies.  In  order  to  establish  a  base¬ 
line  and  measure  the  effect  of  change,  the 
pathfinder  members  must  coUect  data. 
Thus,  appropriate  data  systems  are  select¬ 
ed  as  tools  to  provide  the  data. 

For  example,  the  Air  Force  decided  to 
initiate  a  Reliability  Pathfinder  to  study 
and  define  the  benefits  of  Item  Unique 
Identification  and  Automated  Identifica¬ 
tion  Technology  (AIT)  in  regards  to  facil¬ 
itating  serial  number  tracking  within  the 
maintenance  processes.  The  pathfinder 
team  members  analyzed  the  available  data 
systems  and  chose  to  use  DRILS  as  the 
tool  with  which  to  coUect  their  data.  They 
performed  their  analysis  on  selected  B-52 
avionics  maintenance  occurring  on  the 
flight  line  and  at  the  depot.  DRILS  was 
used  basically  as  is  with  a  few  minor  soft¬ 
ware  modifications  to  facilitate  specific 
data  collection  and  analysis.  The  result  is 
that  the  Air  Force  Reliability  Pathfinder 
has  proven  very  successful.  Reports  are 


currently  being  prepared  that  have  the 
potential  to  positively  impact  the  future  of 
serialized  asset  tracking. 

What  I  learned  while  participating  in 
three  Air  Force  and  two  joint  Air  Force 
and  Army  service  pathfinder  initiatives  is 
that  they  are  useful  for  unifying  a  vision. 
Headquarters  depends  on  field  users  to 
define  requirements  for  them.  Users  want 
headquarters  to  make  decisions  and 
investments  that  wiU  improve  their  work 
environment,  but  often  do  not  know  how 
to  effectively  communicate  requirements. 
I  saw  disconnects  occurring  on  both  sides. 

pathfinder  initiative 
provides  an  excellent 
forum  ...  to  collaborate  in 
a  closed  environment, 
reach  a  common 
understanding,  solve 
longstanding  issues,  and 
communicate  those 
solutions  to  all  parties.  If 
you  can  team  with  an 
existing  legacy  system  ... 
then  you  have  significantly 
increased  your  odds  of 
being  approved.*^ 

A  disconnect  may  occur  in  the  under¬ 
standing  of  the  big  picture  at  the  user 
level,  while  the  headquarters  may  not 
completely  understand  the  detailed  needs 
of  the  user.  A  pathfinder  initiative  pro¬ 
vides  an  excellent  forum  for  these  two 
groups  to  collaborate  in  a  closed  environ¬ 
ment,  reach  a  common  understanding, 
solve  longstanding  issues,  and  communi¬ 
cate  those  solutions  to  all  parties. 

To  participate  in  a  pathfinder,  you 
need  to  apply  your  gap  assessment,  the 
backing  and  breadth  of  your  champions, 
and  your  legacy  system  comparisons  to 
make  your  case  to  headquarters  of  how 
you  can  help  with  a  pathfinder  effort.  If 
you  can  team  with  an  existing  legacy  sys¬ 
tem  to  solve  the  pathfinder  needs,  then 
you  have  significantly  increased  your  odds 
of  being  approved. 


Pathfinder  efforts  are  as  resource-chal¬ 
lenged  as  any  other  program.  Therefore, 
financial  resources  to  support  your  efforts 
will  be  very  limited.  However,  the  expo¬ 
sure  and  lessons  learned  from  a  successful 
pathfinder  effort  are  significant. 
Pathfinder  progress  reports  are  reviewed 
at  the  highest  management  levels.  A  suc¬ 
cessful  pathfinder  effort  wiU  often  lead  to 
other  pathfinders  that  increase  the  expo¬ 
sure  of  your  system  as  well  as  your  accep¬ 
tance  within  the  DoD  IT  community. 

Do  You  Have  Portal 
Capability? 

How  many  user  names  and  passwords  do 
you  currently  maintain?  Do  you  think 
users  will  be  willing  to  add  your  system  to 
the  list  as  well?  I  am  personally  aware  of 
an  office  that  did  a  Lean  study  and  found 
they  lost  1.5  hours  of  productivity  per 
day  logging  in  and  out  of  22  data  systems 
to  do  their  job.  This  frustrated  the  work¬ 
ers  and  decreased  their  overall  job  satis¬ 
faction. 

You  can  increase  your  probability  of 
user  acceptance  by  checking  to  see  if 
there  is  a  portal  such  as  the  Air  Force 
Portal  or  Army  Knowledge  Online 
(AKO)  that  you  can  integrate  with  to 
provide  streamlined  sign-on  capability. 
Check  with  your  portal  for  specific 
requirements.  Interfacing  with  a  portal 
will  also  demonstrate  your  ability  to  inte¬ 
grate  within  the  enterprise,  and  decision 
makers  will  often  sway  your  way  when 
compared  to  a  non-integrated  system. 

In  the  case  of  DRILS,  we  were  able  to 
integrate  the  application  with  the  Air 
Force  Portal  that  streamlined  sign-on  for 
many  of  our  DoD  users.  It  also  allowed  us 
to  extend  use  of  the  application  to  select¬ 
ed  F-16  DoD  repair  contractors  in  the 
.com  world  that  facilitated  increased  visi¬ 
bility  of  F-16  avionics  repairs  worldwide. 

The  DRILS  Air  Force  Portal  integra¬ 
tion  went  fairly  smoothly  with  only  rela¬ 
tively  minor  edits  to  our  authentication 
process.  We  did  encounter  policy  chal¬ 
lenges  that  we  felt  had  to  be  overcome. 
We  had  several  hundred  users  who 
depended  on  the  application  for  depot 
production.  If  the  portal  went  offline,  we 
risked  not  collecting  valuable  data  as  pro¬ 
duction  would  continue,  but  data  capture 
may  not  catch  up.  Thus,  we  still  wanted 
our  users  to  be  able  to  access  the  system. 
The  Air  Force  Portal  policy  was  that  our 
application  authentication  must  be 
restricted  to  portal  users  and  deny  direct 
access  and  login  via  our  non-portal  Web 
address.  It  took  a  few  e-mails  and  confer¬ 
ence  calls  as  well  as  a  formal  waiver 
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request  to  gain  approval  for  a  hybrid 
security  model  that  would  allow  both  Air 
Force  Portal  and  manual  authentication. 
This  allowed  us  to  ensure  maximum 
availability  to  at  least  our  .mil  users. 
Those  in  the  .com  world  would  have  to 
remain  dependent  on  the  Air  Force 
Portal  availability. 

Have  You  Done  Your 
Paperwork? 

I  dislike  doing  paperwork  as  much  as  the 
next  person.  Unfortunately,  paperwork  is 
just  part  of  the  territory  when  it  comes  to 
building  and  fielding  a  DoD  IT  system. 
Recent  NDAA  legislation  leaves  you  with 
little  choice.  You  risk  incurring  stiff  finan¬ 
cial  and  judicial  penalties  if  you  do  not 
complete  your  paperwork. 

The  following  questions  address  the 
two  main  documentation  areas  that  should 
be  common  across  the  DoD.  Each  agency 
may  impose  additional  requirements.  You 
will  need  to  check  with  your  respective 
agency  for  details. 

Are  You  Registered  With  PfM? 

PfM  is  your  required  first  approval  stop 
for  any  local  or  global  data  system.  PfM  is 
the  management  of  selected  groupings  of 
investments  using  integrated  strategic 
planning,  integrated  architectures,  perfor¬ 
mance  measures,  risk-management  tech¬ 
niques,  transition  plans,  and  portfolio 
investment  strategies.  The  PfM  process  is 
driven  by  a  number  of  legislative  acts  and 
DoD  directives. 

At  the  root  of  PfM  is  the  Clinger- 
Cohen  Act  of  1996  that  requires  agencies 
to  use  a  capital  planning  and  investment 
control  process  to  provide  for  selection, 
management,  and  evaluation  of  IT 
investments  [8].  Consequently,  the  DoD 
published  Directive  8115.01,  Infor¬ 
mation  Technology  Portfolio  Manage¬ 
ment,  to  establish  policy  and  assign 
responsibility  for  the  management  of 
DoD  IT  investments  as  portfolios  that 
focus  on  improving  DoD  capabilities  and 
mission  outcomes  [9].  DoD  Directive 
8000.1,  Management  of  DoD  Infor¬ 
mation  Resources  and  Information 
Technology,  establishes  the  requirement 
for  a  Chief  Information  Officer  (CIO) 
role  in  the  agencies  to  manage  these  port¬ 
folios.  The  CIOs  designate  portfolio 
managers  to  manage  their  portfolios  [10]. 
Portfolio  managers  interact  with  DoD  IT 
system  program  managers  to  report  the 
status  of  their  programs. 

The  DoD  Enterprise  Information 
Technology  Portfolio  Repository 
(DITPR)  is  one  system  used  to  track 


portfolios.  DITPR  was  selected  by  the 
DoD  CIO  as  the  enterprise  shared  space 
for  IT  PfM  data  for  all  DoD  business  IT 
systems.  However,  each  branch  has  its 
own  methods  of  IT  registry  that  feed  to 
DITPR.  The  Air  Force  uses  the 
Enterprise  Information  Technology  Data 
Repository  (EITDR),  the  Navy  and 
Marines  use  the  DITPR-DON 
(Department  of  Navy)  system,  and  the 
Army  uses  the  Army  Portfolio 
Management  System  (APMS)  as  their 
registry.  All  of  these  systems  are  used  to 
record  investment  review  and  certifica¬ 
tion  submission  information.  Federal 
Information  Security  Management  Act 
(FISMA)  of  2002  assessments,  and  more 
[1 1].  The  IT  Lean  acquisition  process  and 
security,  interoperability,  supportability, 
sustainability,  and  usability  processes  are 
integrated  into  these  systems  as  well. 

These  systems  are  necessary  in  order 
to  provide  portfolio  managers  access  to 
information  needed  to  do  the  following: 
maximize  value  of  IT  investments  while 
minimizing  risk,  improve  communication 
and  alignment  between  IT  and  DoD 
leaders,  facilitate  team  thinking  versus 
individual  commands  or  units,  enable 
more  efficient  use  of  assets,  reduce  the 
number  of  redundant  projects  and  elimi¬ 
nate  non-value  added  projects,  and  sup¬ 
port  an  enterprise  IT  investment 
approach. 

Portfolio  managers  depend  on  IT 
program  managers  to  keep  the  portfolio 
data  current  for  their  respective  systems 
in  order  to  fulfill  their  goals.  Participation 
in  PfM  is  not  an  option.  There  are  serious 
consequences  for  not  complying  with  all 
of  the  PfM  requirements. 

If  you  are  building  a  new  system  or 
expanding  the  capability  of  an  existing 
system,  you  must  submit  a  capability 
request  to  your  respective  portfolio  man¬ 
ager  to  get  authorization.  This  is  where 
you  once  again  tap  into  your  foundation¬ 
al  data  that  describes  the  gaps  you  are  fill¬ 
ing.  You  may  need  to  arrange  a  meeting 
with  your  portfolio  manager  to  describe 
why  you  need  the  capability  and  that  a 
similar  capability  does  not  exist.  The 
portfolio  manager  has  to  weigh  a  lot  of 
criteria  when  making  a  decision  to  autho¬ 
rize  your  request  and  may  request  addi¬ 
tional  data  to  reach  their  decision.  You 
may  even  be  invited  to  a  ^  before  a 
board  who  is  evaluating  similar  systems 
within  the  portfolio. 

Do  You  Meet  the  Information 
Assurance  Requirements? 

One  of  the  first  things  that  a  portfolio 


manager  will  evaluate  is  whether  you  com¬ 
ply  with  mandatory  information  assurance 
(lA)  requirements  for  your  system.  lA  is 
more  important  today  than  it  ever  has 
been;  information  warfare  attacks  are  a 
reality.  FISMA  requires  each  federal 
agency  to  develop,  document,  and  imple¬ 
ment  an  agency-wide  program  to  provide 
information  security  for  the  information 
and  information  systems  that  support  the 
operations  and  assets  of  the  agency  [11]. 

In  order  to  comply  with  FISMA 
requirements,  the  DoD  has  created  the 
Defense  Information  Assurance  Certi¬ 
fication  and  Accreditation  Process  (DIA- 
CAP)  that  replaced  the  Defense  Infor¬ 
mation  Technology  Security  Certification 
and  Accreditation  Process.  DIACAP 
assigns,  implements,  and  validates  DoDI 
8500.2  standardized  lA  controls  and  man¬ 
ages  lA  posture  across  DoD  information 
systems  consistent  with  FISMA  legislative 
policy  as  wed  as  DoD  regulatory  policy 
found  in  the  8500  series  of  directives  [12]. 

You  need  to  ensure  that  your  system 
remains  compliant  with  the  lA  require¬ 
ments  identified  in  the  DoD  lA  8500 
series  of  directives.  If  you  do  not,  then 
you  will  not  be  authorized  to  operate  on 
the  DoD  network  or  interface  to  legacy 
systems.  DoDI  8500.2  assigns  lA  controls 
to  three  Mission  Assurance  Categories 
(MAC)  and  three  data  sensitivity  levels. 
You  will  need  to  evaluate  your  system  and 
select  one  MAC  and  one  data  sensitivity 
level  appropriate  to  your  system  and  mis¬ 
sion  that  win  determine  what  your  lA  con¬ 
trol  requirements  are. 

Once  you  have  shown  that  you  comply 
with  the  lA  control  requirements,  you 
must  submit  a  Certification  and 
Accreditation  (C&A)  package  to  your 
Designated  Approval  Authority  for 
approval  using  the  DIACAP  workflow. 
When  your  package  is  approved,  an 
Authority  to  Operate  will  be  issued  that  is 
valid  for  three  years  from  the  date  it  is 
issued.  DIACAP  also  requires  annual 
security  reviews  of  the  C&A  package  and 
those  reviews  are  reported  to  the  portfolio 
manager  through  the  appropriate  portfo¬ 
lio  registry  such  as  EITDR,  DITPR- 
DON,  or  APMS. 

Complying  with  mandatory  lA 
requirements  is  just  one  piece  of  the  secu¬ 
rity  puzzle.  You  need  to  also  ensure  the 
system  is  programmed  defensively  using 
secure  coding  techniques  to  ensure  that 
your  system  and  its  information  are  not 
compromised.  Web  application  security  is 
considered  a  weak  point  in  an  IT  security 
wall  and  subject  to  information  warfare 
attack. 
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Coming  Events 


March  3-7 

SD  West  2008  Software  Development 
Conference  and  Expo  West 
Santa  Clara,  CA 

http://sdexpo.conn 

March  4-5 

Warfighters  Vision  2008 
Tampa,  FL 

www.afei.org 

March  11-12 

2008  Military  and  Aerospace 
Electronics  Eorum 
San  Diego,  CA 

http://nntc08.events.pennnet.conn/ 

fl//index.cfnn 

March  16-18 

2008  Engineering  Research  Council 
Summit,  Workshop  and  Eorum 
Arlington,  VA 

www.asee.org/conferences/erc/2008/ 

index.cfm. 

March  17-20 

2008  SEPG 
Tampa,  FL 

www.sei.cmu.edu/sepg 

March  18-20 

Sea  -  Air  -  Space  2008 
Washington,  D.C. 
www.sasexpo.org/2008 

March  24-28 

International  Testing  Certification 
Super  Week 
Chicago,  IL 

www.testinginstitute.com 

April  29-May  2 

&  Sofhimte 

2008  Systems  and  Software 
Technology  Conference 
Las  Vegas,  NV 

www.sstc-online.org 


Coming  Events:  Please  submit  coming  events  that 
are  of  interest  to  our  readers  at  least  90  days 
before  registration.  E-mail  announcements  to: 
nicole.kentta@hill.af.mil. 


Agency  has  published  a  Security  Technical 
Implementation  Guide  titled  “Application 
Security  and  Development  Security.”  This 
can  be  downloaded  at  <http://iase. 
disa.mil>. 

Conclusion 

It  is  possible  to  grow  a  local  product  into 
an  enterprise  system  with  today’s 
increased  PfM  and  lA  requirements.  We 
started  DRILS  in  July  of  2000,  delivered 
our  rapid  prototype  in  September  of 
2000,  and  used  evolutionary  develop¬ 
ment  from  that  point  forward.  During 
my  six  years  as  chief  architect,  I  saw  a  lot 
of  transformation  on  how  IT  systems  are 
certified  and  supported.  I  am  glad  to  say 
that  it  is  becoming  less  of  a  paperwork 
drill  now.  However,  there  are  stiU  a  lot  of 
steps  to  be  checked  off  You  still  have  to 
do  your  homework  and  some  paperwork 
to  lay  your  foundation  in  order  to  educate 
your  user  community,  champions,  and 
portfolio  managers  on  why  your  system 
should  exist. 

Align  yourself  wherever  possible  with 
the  goals,  objectives,  and  standards  of 
your  agency  and  the  DoD  in  general. 
Pathfinders  are  an  excellent  avenue  to 
prove  your  alignment,  increase  your  visi¬ 
bility,  and  gain  acceptance  at  the  head¬ 
quarters  level.  You  can  further  demon¬ 
strate  your  capability  to  integrate  in  the 
enterprise  by  facilitating  streamlined 
sign-on  to  your  application  through  a 
portal  such  as  the  Air  Force  portal  or 
AKO. 

Getting  integrated  into  the  DoD  enter¬ 
prise  is  not  only  a  technology  challenge  but 
also  a  challenge  of  navigating  the  approval 
process.  However,  with  the  proper  prepa¬ 
ration  the  approval  process  will  be  much 
easier  to  navigate  success  fully.  ♦ 
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Capers  Jones 
Software  Productivity  Kesearch,  LUS 

because  large  software  projects  are  troublesome  and  often  out  get  out  of  control,  a  number  of  effective  development  methods 
have  been  created  to  help  reduce  the  problems  of  large  software  projects.  Two  of  these  methods  include  the  well-known 
Capability  Maturity  ModeF  (CMM®)  and  the  newer  CALM  Integration^^  (CALMI®)  of  the  Software  Engineering  Institute 
(SEI).  When  these  methods  are  applied  to  small  projects,  it  is  usually  necessary  to  customi^  them  because,  in  their  original 
form,  they  are  somewhat  cumbersome  for  small  projects,  although  proven  effective  for  large  applications.  More  recently,  alter¬ 
nate  methods  derived  from  smaller  software  projects  have  become  popular  under  the  general  name  of  Agile  software  develop¬ 
ment.  Since  the  Agile  methods  originated  for  fairly  small projects,  they  can  be  applied  without  customisation  and  often  return 
very  positive  results.  It  is  possible  to  merge  Agile  concepts  with  CMM  and  CMMI  concepts,  but  projects  cited  in  this  article 
did  not  do  so  because  such  mergers  are  comparatively  rare. 


Software  applications  come  in  a  wide 
range  of  sizes  and  types.  Each  size 
and  type  tends  to  have  evolved  typical 
development  practices.  For  example, 
military  projects  are  usually  much  more 
formal  and  perform  many  more  over¬ 
sight  and  control  activities  than  civilian 
projects.  Systems  software  tends  to  have 
more  kinds  of  testing  and  quality  con¬ 
trol  activities  than  information  systems. 

When  application  sizes  are  consid¬ 
ered,  there  are  very  significant  differ¬ 
ences  in  the  development  processes  that 
are  most  commonly  used  for  large  soft¬ 
ware  projects  compared  to  small  soft¬ 
ware  projects. 

Before  discussing  size  differences,  it 
is  best  to  start  by  defining  what  is  meant 
by  the  terms  large  and  small.  Although 
there  is  no  agreed-to  definition  for  these 
terms,  the  author  generally  regards  large 
applications  as  being  those  whose  size 
exceeds  10,000  function  points  or  about 
1,000,000  source  code  statements.  The 
author  regards  small  applications  as 
those  whose  size  is  at  or  below  1,000 
function  points  or  about  100,000  source 
code  statements. 

It  is  a  significant  observation  that  out 
of  16  software  lawsuits  where  the  author 
served  as  an  expert  witness,  15  of  them 
were  for  applications  in  the  10,000  func¬ 
tion  point  size  range,  or  larger.  The 
smallest  application  noted  during  litiga¬ 
tion  was  3,000  function  points. 

Usually  software  applications  of 
1,000  function  points  or  below  are  fairly 
trouble-free  and,  therefore,  seldom  end 
up  in  litigation  for  outright  failure,  qual¬ 
ity  problems,  cost  overruns,  or  schedule 

®  2007  by  Capers  Jones.  All  rights  reserved. 

®  Capability  Maturity  Model,  CMM  and  CMMI  are  regis¬ 
tered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie 
Mellon  University. 

CMM  Integration,  Team  Software  Process,  Personal 
Software  Process,  TSP,  and  PSP  are  service  marks  of 
Carnegie  Mellon  University. 


slips.  On  the  other  hand,  an  alarming 
percentage  of  applications  larger  than 
10,000  function  points  exhibit  serious 
quality  problems,  outright  failure,  or 
massive  overruns. 

Some  readers  might  think  that  1,000 
function  points  or  100,000  source  code 
statements  are  still  rather  large. 
However,  there  are  very  few  commercial 
software  applications  or  business  appli- 

^y\lhen  application  sizes 
are  considered,  there 
are  very  significant 
differences  in  the 
development  processes 
that  are  most  commonly 
used  for  large  software 
projects  compared  to 
small  software  projects. 

cations  that  are  smaller  than  this  size 
range.  Software  in  the  range  of  100 
function  points  or  10,000  source  code 
statements  usually  consists  of  enhance¬ 
ments  to  larger  applications  rather  than 
stand-alone  applications.  Even  the 
smallest  commercial  off-the-shelf  soft¬ 
ware  packages  are  in  the  range  of  1,000 
function  points. 

The  overall  size  range  of  software 
applications  noted  by  the  author  runs 
from  a  high  of  about  300,000  function 
points  (30,000,000  source  code  state¬ 
ments)  down  to  about  0.1  function 
points  (10  source  code  statements)  for  a 
small  bug  repair. 


At  the  extreme  high  end  are  very  few 
massive  applications  such  as  enterprise 
resource  planning  (ERP)  packages  and 
some  large  defense  systems.  Operating 
systems  and  major  application  packages 
such  as  Microsoft  Office  are  in  the  range 
of  100,000  function  points  or 
10,000,000  source  code  statements. 
Various  components  of  Microsoft 
Office  such  as  Excel,  Word,  PowerPoint, 
etc.,  are  between  5,000  and  10,000  func¬ 
tion  points  in  size. 

Origins  of  the  CMM  and 
Agile  Development 

The  SEI  was  incorporated  in  1984  and  is 
located  at  Carnegie  Mellon  University  in 
Pittsburgh,  Pennsylvania.  The  SEI  was 
originally  funded  by  the  Defense 
Advanced  Research  Projects  Agency. 

Some  of  the  major  concerns  that  led 
to  the  creation  of  the  SEI  were  the  very 
serious  and  common  problems  associat¬ 
ed  with  large  software  projects.  One  of 
the  early,  major,  and  continuing  activities 
of  the  SEI  was  the  development  of  a 
formal  evaluation  or  assessment  schema 
for  evaluating  the  capabilities  of  defense 
contractors,  which  was  termed  CMM. 
Watts  Humphrey’s  book  on  this  topic, 
‘‘Managing  the  Software  Process” 
became  a  bestseller  [1]. 

The  initial  version  of  the  CMM  was 
published  by  SEI  in  1987,  and  continued 
to  grow  and  evolve  for  about  10  years. 
Development  of  the  CMM  more  or  less 
stopped  in  1997  and  the  emphasis 
switched  to  the  next  generation,  or  the 
CMMI.  The  CMMI  was  initially  pub¬ 
lished  in  2002  and  has  been  updated  sev¬ 
eral  times.  Associated  with  the  CMMI 
are  Watts  Humphrey’s  Team  Software 
Process^^  (TSP^^  and  Personal  Software 
Process'^  (PSP'"0- 

The  SEI  assessment  approach  is  now 
www.stsc.hill.af.mil  9 
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well  documented  and  well  covered  in 
software  literature.  Indeed,  Addison 
Wesley  Longman  has  an  entire  series 
devoted  to  SEI  assessments.  The  SEI 
assessment  data  is  collected  by  means  of 
on-site  interviews  using  both  a  standard 
questionnaire  and  also  observations  and 
informal  data. 

Because  of  the  importance  of  very 
large  systems  to  the  Department  of 
Defense,  the  SEI  assessment  approach 
originally  dealt  primarily  with  the  soft¬ 
ware  processes  and  methodologies  used 
by  large  companies  that  produced  large 
systems.  The  original  SEI  assessment 
approach  was  derived  from  the  best 
practices  used  by  leading  corporations 
such  as  IBM  and  ITT,  which  employ 
from  5,000  to  more  than  50,000  soft¬ 
ware  professionals,  and  which  could 
safely  build  systems  in  excess  of 
1,000,000  lines  of  code  or  10,000  func¬ 
tion  points. 

Based  on  the  patterns  of  answers  to 
the  SEI  assessment  questions,  the  final 
result  of  the  SEI  assessment  process 
places  the  software  organization  on  one 
of  the  levels  of  a  five-point  maturity 
scale.  The  five  plateaus  of  the  SEI  matu¬ 
rity  levels  are  shown  in  Table  1 . 

It  is  immediately  obvious  that  the 
distribution  of  software  organizations  is 
skewed  toward  the  low  end  of  the  scale. 
A  similar  kind  of  skew  would  occur  if 
you  were  to  look  at  the  distribution  of 
candidates  selected  to  enter  the 
Olympics  for  events  such  as  downhill 
skiing.  Most  ordinary  citizens  could  not 
qualify  at  all.  Very  few  athletes  could 
make  it  to  the  Olympic  tryouts,  even 
fewer  would  represent  their  countries, 
and  only  three  athletes  from  around  the 
world  will  win  medals  in  each  event. 

As  data  is  collected,  it  becomes  evi¬ 
dent  that  there  is  quite  a  bit  of  overlap 
among  the  various  SEI  maturity  levels. 
For  example,  in  terms  of  both  quality 
and  productivity,  the  best  software  pro¬ 
jects  from  Level  1  organizations  can  be 
superior  to  the  worst  developed  by  Level 
3  organizations,  although  the  statistical 
averages  of  Level  3  are  far  better  than 
those  of  Level  1. 

There  is  now  fairly  solid  evidence 
about  the  CMM  from  many  studies. 


When  organizations  move  from  CMM 
Level  1  up  to  Level  2,  3,  4,  and  5,  their 
productivity  and  quality  levels  tend  to 
improve  based  on  samples  at  each  level. 

As  of  2007,  the  newer  CMMI  has 
less  empirical  data  than  the  older  CMM, 
which  is  not  surprising  given  its  more 
recent  publication  date.  However,  the 
TSP  and  PSP  methods  do  have  enough 
data  to  show  that  they  are  successful  in 
improving  both  quality  and  productivity 
at  the  same  time. 

When  the  CMM  originated  in  the 
1980s,  the  waterfall  method  of  develop¬ 
ment  was  the  most  common  at  that  time 
and  was  implicitly  supported  by  most 
companies  that  used  the  early  CMM. 
However,  other  methods  such  as  spiral, 

the  CMM 

provided  was  a  solid 
framework  of  activities, 
much  better  rigor  in 
the  areas  of  quality 
control  and  change 
management,  and  much 
better  measurement  of 
progress,  quality,  and 
productivity  than  was 
previously  the  norm/^ 

iterative,  etc.,  were  quickly  included  in 
the  CMM  as  well.  The  CMM  is  neutral  as 
to  development  methods,  but  among  the 
author’s  clients  who  adopted  the  CMM, 
about  80  percent  also  used  the  waterfall 
method. 

What  the  CMM  provided  was  a  solid 
framework  of  activities,  much  better 
rigor  in  the  areas  of  quality  control  and 
change  management,  and  much  better 
measurement  of  progress,  quality,  and 
productivity  than  was  previously  the 
norm. 


The  history  of  the  Agile  methods  is 
not  as  clear  as  the  history  of  the  CMM 
because  the  Agile  methods  are  some¬ 
what  diverse.  However,  in  2001  the 
famous  Agile  Manifesto  was  published 
[2].  This  provided  the  essential  princi¬ 
ples  of  Agile  development.  That  being 
said,  there  are  quite  a  few  Agile  varia¬ 
tions  including  eXtreme  Programming 
(XP),  Crystal  Development,  Adaptive 
Software  Development,  Feature  Driven 
Development,  and  several  others. 

Some  of  the  principle  beliefs  found 
in  the  Agile  Manifesto  include  the  fol¬ 
lowing: 

•  Working  software  is  the  goal,  not 
documents. 

•  Working  software  is  the  principle 
measure  of  success. 

•  Close  and  daily  contact  between 
developers  and  clients  is  necessary. 

•  Face-to-face  conversation  is  the  best 
form  of  communication. 

•  Small,  self-organizing  teams  give  the 
best  results. 

•  Quality  is  critical,  so  testing  should 
be  early  and  continuous. 

The  Agile  methods,  the  CMM,  and 
the  CMMI  are  all  equally  concerned 
about  three  of  the  same  fundamental 
problems: 

1.  Software  requirements  always 
change. 

2.  Fixing  software  bugs  is  the  most 
expensive  software  activity  in  history. 

3.  High  quality  leads  to  high  productiv¬ 
ity  and  short  schedules. 

However,  the  Agile  method  and  the 

CMM/ CMMI  approach  draw  apart  on 
two  other  fundamental  problems: 

1 .  Paperwork  is  the  second  most  expen¬ 
sive  software  activity  in  history. 

2.  Without  careful  measurements,  con¬ 
tinuous  progress  is  unlikely. 

The  Agile  method  takes  a  strong 
stand  that  paper  documents  in  the  form 
of  rigorous  requirements  and  specifica¬ 
tions  are  too  slow  and  cumbersome  to 
be  effective.  In  the  Agile  view,  daily 
meetings  with  clients  are  more  effective 
than  written  specifications.  In  the  Agile 
view,  daily  team  meetings  or  Scrum  ses¬ 
sions  are  the  best  way  of  tracking 
progress,  as  opposed  to  written  status 
reports.  The  CMM  and  CMMI  do  not 
fully  endorse  this  view. 

The  CMM  and  CMMI  take  a  strong 
stand  that  measurements  of  quality,  pro¬ 
ductivity,  schedules,  costs,  etc.,  are  a  nec¬ 
essary  adjunct  to  process  improvement 
and  should  be  done  well.  In  the  view  of 
the  CMM  and  CMMI,  it  is  hard  to  prove 
that  a  methodology  is  a  success  or  not 
without  data  that  demonstrates  effective 


Table  1 :  Fm  Levels  of  the  CMM 


SEI  Maturity  Level 

Meaning 

Frequency  of  Occurrence 

1  =  Initial 

Chaotic 

75.0% 

2  =  Repeatable 

Marginal 

15.0% 

3  =  Defined 

Adequate 

8.0% 

4  =  Managed 

Good  to  excellent 

1 .5% 

5  =  Optimizing 

State  of  the  art 

0.5% 
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progress.  The  Agile  method  does  not 
fully  endorse  this  view.  In  fact,  one  of 
the  notable  gaps  in  the  Agile  approach  is 
any  quantitative  quality  or  productivity 
data  that  can  prove  the  success  of  the 
methods. 

Differences  in  Development 
Activities  Between  the  Agile 
and  CMM/CMMI  Methods 

In  many  industries,  building  a  large  prod¬ 
uct  is  not  the  same  as  building  a  small 
product.  Consider  the  differences  in  spe¬ 
cialization  and  methods  required  to  build 
a  wooden  kayak  versus  building  an 
80,000-ton  cruise  ship.  A  kayak  can  be 
constructed  by  a  single  individual  using 
only  hand  tools,  but  a  large,  modern 
cruise  ship  requires  more  than  500  work¬ 
ers  including  specialists  such  as  pipe  fit¬ 
ters,  electricians,  steel  workers,  welders, 
painters,  interior  decorators,  air  condi¬ 
tioning  specialists,  and  many  others. 

Software  follows  a  similar  pattern: 
Building  a  large  system  in  the  10,000 
function  point  range  is  more  or  less 
equivalent  to  building  other  large  struc¬ 
tures  such  as  ships,  office  buildings,  or 
bridges.  Many  kinds  of  specialists  are 
utilized  and  the  development  activities 
are  quite  extensive  compared  to  smaller 
applications. 

Because  the  CMM  approach  was 
developed  in  the  1980s  when  the  water¬ 
fall  method  was  common,  it  is  not  diffi¬ 
cult  to  identify  the  major  activities  that 
are  typically  performed.  For  an  applica¬ 
tion  of  1,000  function  points  (approxi¬ 
mately  100,000  source  code  statements), 
the  following  20  normal  activities  were 
noted  among  the  author’s  clients  who 
used  either  the  CMM  or  CMMI: 

1 .  Requirements. 

2.  Prototyping. 

3.  Architecture. 

4.  Project  planning  and  estimating. 

5.  Initial  design. 

6.  Detailed  design. 

7.  Design  inspections. 

8.  Coding. 

9.  Reuse  acquisition. 

10.  Code  inspections. 

1 1 .  Change  and  configuration  control. 

12.  Software  quality  assurance. 

13.  Integration. 

14.  Test  plans. 

15.  Unit  testing. 

16.  New  function  testing. 

17.  Regression  testing. 

18.  Integration  testing. 

19.  Acceptance  testing. 

20.  Project  management. 

Using  the  CMM  and  CMMI,  the 


Successful  Hybrid  Approaches  to 
Software  Development 

One  of  the  major  trends  in  the  industry  since  about  1997  has  been  to  couple  the  most 
effective  portions  of  various  software  development  methodologies  to  create  hybrid 
approaches.  These  hybrid  methods  are  often  successful  and  often  achieve  higher  qual¬ 
ity  and  productivity  rates  than  the  original  pure  methods.  As  of  2007  some  interesting 
examples  of  the  hybrid  approach  include,  in  alphabetical  order: 

•  Agile  joined  with  object-oriented  development  (OOD). 

•  Agile  joined  with  Lean  Six-Sigma. 

•  Agile  joined  with  TSP  and  PSP. 

•  Agile  joined  with  the  CMM  and  CMMI. 

•  CMM  and  CMMI  joined  with  Six-Sigma. 

•  CMM  and  CMMI  joined  with  International  Standardization  for  Organization  (ISO) 
standard  certification. 

•  CMM  and  CMMI  joined  with  Information  Technology  Infrastructure  Library  (ITIL). 

•  CMM  and  CMMI  joined  with  Ouality  Function  Deployment  (OFD). 

•  CMM  and  CMMI  joined  with  TSP  and  PSP. 

•  XP  joined  with  Lean  Six-Sigma. 

•  ITIL  joined  with  Six-Sigma. 

•  ISO  joined  with  TickIT 

•  OOD  joined  with  service-oriented  architecture  (SOA). 

•  Six-Sigma  joined  with  OFD. 

•  Six-Sigma  joined  with  ITIL. 

•  Six-Sigma  joined  with  TSP/PSP. 

•  SOA  joined  with  ITIL. 

•  TickIT  joined  with  Six-Sigma. 

The  list  above  only  links  pairs  of  development  methods  that  are  joined  together.  From 
time  to  time  three  or  even  four  or  five  development  practices  have  been  joined  togeth¬ 
er: 

•  Agile  joined  with  OOD,  joined  with  Lean  Six-Sigma,  and  joined  with  ITIL. 

•  CMM  joined  with  Agile,  joined  with  TSP/PSP,  joined  with  Six-Sigma,  and  joined  with 

OFD. 


entire  application  of  1,000  function 
points  would  have  the  initial  require¬ 
ments  gathered  and  analyzed,  the  speci¬ 
fications  written,  and  various  planning 
document  produced  before  coding  got 
under  way. 

By  contrast,  the  Agile  method  of 
development  would  follow  a  different 
pattern.  Because  the  Agile  goal  is  to 
deliver  running  and  usable  software  to 
clients  as  rapidly  as  possible,  the  Agile 
approach  would  not  wait  for  the  entire 
1,000  function  points  to  be  designed 
before  coding  started. 

What  would  be  most  likely  with  the 
Agile  methods  would  be  to  divide  the 
overall  project  into  four  smaller  projects, 
each  of  about  250  function  points  in 
size.  In  Agile  terminology,  these  smaller 
segments  are  termed  iterations  or  some¬ 
times  sprints. 

However,  in  order  to  know  what  the 
overall  general  set  of  features  would  be, 
an  Agile  project  would  start  with  Iteration 
0  OT  2i  general  planning  and  require¬ 
ments -gathering  session.  At  this  session, 
the  users  and  developers  would  scope 
out  the  likely  architecture  of  the  applica¬ 
tion  and  then  subdivide  it  into  a  number 
of  iterations. 

Also,  at  the  end  of  the  project  when 


all  of  the  iterations  have  been  complet¬ 
ed,  it  will  be  necessary  to  test  the  com¬ 
bined  iterations  at  the  same  time. 
Therefore,  a  release  phase  follows  the 
completion  of  the  various  iterations.  For 
the  release,  some  additional  documenta¬ 
tion  may  be  needed.  Also,  cost  data  and 
quality  data  need  to  be  consolidated  for 
all  of  the  iterations.  A  typical  Agile 
development  pattern  might  resemble  the 
following: 

•  Iteration  0 

1.  General  overall  requirements. 

2.  Planning. 

3.  Sizing  and  estimating. 

4.  Funding. 

•  Iterations  1-4 

1.  User  requirements  for  each  itera¬ 
tion. 

2.  Test  planning  for  each  iteration. 

3.  Testing  case  development  for 
each  iteration. 

4.  Coding. 

5.  Testing. 

6.  Scrum  sessions. 

7.  Iteration  documentation. 

8.  Iteration  cost  accumulation. 

9.  Iteration  quality  data. 

•  Release 

1 .  Integration  of  all  iterations. 

2.  Final  testing  of  all  iterations. 
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Agile 

CMM  Level  3 

Difference 

Size  in  function  points 

1,000 

1,000 

0 

Size  in  Java  code  statements 

50,000 

50,000 

0 

Monthly  burdened  cost 

$7,500 

$7,500 

0 

Work  hours  per  month 

132 

132 

0 

Project  staff 

5 

7 

2 

Project  effort  (months) 

66 

115 

49 

Project  effort  (hours) 

8,712 

15,180 

6,486 

Project  schedule  (months) 

14 

19 

5 

Project  cost 

$495,000 

$862,500 

$367,500 

Function  points  per  month 

15.15 

8.67 

-6.46 

Work  hours  per  function  point 

8.71 

15.18 

6.47 

LOC  per  month 

758 

435 

-323 

Function  point  assignment  scope 

200 

143 

-57 

LOC  assignment  scope 

10,000 

7,143 

-2,857 

Cost  per  function  point 

$495 

$863 

$368 

Cost  per  LOC 

$9.90 

$17.25 

$7.35 

Defect  potential 

4,250 

4,500 

250 

Defect  potential  per  function  point 

4.25 

4.50 

0.25 

Defect  removal  efficiency 

90% 

95% 

0.5% 

Delivered  defects 

425 

225 

-200 

High-severity  defects 

128 

68 

-60 

Delivered  defects  per  function  point 

0.43 

0.23 

0.20 

Delivered  defects  per  thousand  LOC 

8.50 

4.50 

-4.00 

Table  2:  Comparison  of  A.gile  and  CMM  Results  for  an  Application  of  1,000  ¥  unction  Points 


3.  Acceptance  testing  of  applica¬ 
tion. 

4.  Total  cost  accumulation. 

5.  Quality  data  accumulation. 

6.  Final  Scrum  session. 

These  are  the  most  interesting  and 
unique  features  of  the  Agile  methods:  1) 
The  decomposition  of  the  application 
into  separate  iterations,  2)  The  daily 
face-to-face  contact  with  one  or  more 
user  representatives,  3)  The  daily  Scrum 
sessions  to  discuss  the  backlog  of  work 
left  to  be  accomplished  and  any  prob¬ 
lems  that  might  slow  down  progress. 
Another  interesting  feature  is  to  create 
the  test  cases  before  the  code  itself  is 
written,  which  is  a  feature  of  XP. 

Comparative  Results  of  Agile 
and  CMM  for  an  Application 
of  1,000  Function  Points 

There  is  much  more  quantitative  data 
available  on  the  results  of  projects  using 
the  CMM  than  for  projects  using  the 
Agile  methods.  In  part  this  is  because 
the  CMM  and  CMMI  include  measure¬ 
ment  as  a  key  practice.  Also,  while  Agile 
projects  do  estimate  and  measure,  some 
of  the  Agile  metrics  are  unique  and  have 
no  benchmarks  available  as  of  2007.  For 
example  some  Agile  projects  use  story 
points,  some  use  running  tested  features, 
some  use  ideal  time,  and  some  measure 
with  use  case  points.  There  are  no  large 
collections  of  data  using  any  of  these 


metrics  nor  are  there  reliable  conversion 
rules  to  standard  metrics  such  as  func¬ 
tion  points. 

For  this  article,  an  artificial  test  bed 
will  be  used  to  examine  the  comparative 
results  of  Agile  and  the  CMM.  The  test 
bed  is  based  on  a  real  application  (an 
online  survey  tool),  but  the  size  has  been 
arbitrarily  set  to  exactly  1,000  function 
points  or  50,000  Java  statements. 

A  sample  of  20  CMM  projects  exam¬ 
ined  by  the  author  and  his  colleagues 
was  condensed  into  an  overall  aggregate 
for  the  CMM  results.  For  the  Agile  data, 
observations  and  discussions  with  prac¬ 
titioners  on  five  projects  were  used  to 
construct  the  aggregate  results.  This  is 
not  a  particularly  accurate  and  reliable 
way  to  perform  comparative  analysis.  As 
it  happens,  none  of  the  20  CMM  appli¬ 
cations  used  Agile  methods,  nor  did  any 
of  the  Agile  projects  use  aspects  of  the 
CMM  or  CMMI. 

Merging  aspects  of  the  Agile  and 
CMM  concepts  is  not  difficult.  Indeed, 
some  CMM  and  CMMI  applications 
circa  2007  do  use  Agile  features  such  as 
Scrum  sessions.  However,  among  the 
author’s  clients,  about  85  percent  of 
CMM/CMMI  users  do  not  use  Agile 
and  about  95  percent  of  Agile  users  do 
not  make  use  of  either  the  CMM  or 
CMMI.  For  example,  at  a  recent  confer¬ 
ence  of  more  than  100  state  software 
managers,  more  than  50  percent  of  cur¬ 
rent  projects  were  using  Agile  methods  — 


but  no  projects  at  all  were  using  either 
the  CMM  or  CMMI. 

Though  briefly  discussed  in  the  side- 
bar,  the  topic  of  hybrid  approaches  is 
not  well  covered  in  the  software  engi¬ 
neering  literature.  Neither  is  the  topic  of 
hybrid  approaches  well  covered  in  terms 
of  benchmarks  and  quantitative  data, 
although  some  hybrid  projects  are  pre¬ 
sent  in  the  International  Software 
Benchmark  Standards  Group  data.  To 
simplify  the  results  in  this  article,  the 
Agile  methods  were  used  in  pure  form  as 
were  the  CMM  and  CMMI  methods. 
Since  the  pure  forms  still  outnumber 
hybrid  approaches  by  almost  10-to-l, 
that  seems  like  a  reasonable  way  to  pre¬ 
sent  the  data. 

The  sizes  of  the  samples  were  not,  of 
course,  exactly  1,000  function  points. 
They  ranged  from  about  900  to  almost 
2,000  function  points.  Not  all  of  the 
samples  used  Java  either.  Therefore  the 
following  comparison  in  Table  2  has  a 
significant  but  unknown  margin  of 
error. 

What  the  comparison  does  provide  is 
side-by-side  data  points  for  a  standard 
test  bed,  with  the  measurements  for 
both  sides  expressed  in  the  same  units  of 
measure.  Hopefully  future  researchers 
from  both  the  Agile  and  CMM/CMMI 
camps  will  be  motivated  to  correct  the 
results  shown  here,  using  better  meth¬ 
ods  and  data. 

Note  that  the  lines  of  code  (LOC)  met¬ 
ric  is  not  reliable  for  economic  studies 
since  it  penalizes  high-level  program¬ 
ming  languages  and  cannot  be  used  for 
cross-language  comparisons.  However, 
since  many  people  still  use  this  metric,  it 
is  provided  here  with  the  caveat  that 
none  of  the  LOC  values  would  be  the 
same  for  other  programming  languages 
such  as  C++,  Smalltalk,  Visual  Basic, 
etc. 

The  data  in  Table  2  is  expressed  in 
terms  of  function  point  metrics  and 
assumes  version  4.2  of  the  counting 
rules  published  by  the  International 
Function  Point  Users  Group. 

The  CMM  version  of  the  project  is 
assumed  to  be  developed  by  an  organi¬ 
zation  at  Level  3  on  the  CMM.  The 
Agile  version  of  the  project  is  assumed 
to  be  developed  by  an  experienced  Agile 
team.  However,  the  Agile  version  is 
assumed  not  to  use  either  the  CMM  or 
CMMI  and  so  has  no  level  assigned. 
This  is  a  reasonable  assumption  because 
very  few  Agile  teams  are  also  CMM  cer¬ 
tified. 

Although  one  comparison  is  proba¬ 
bly  insufficient,  observations  of  many 
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small  projects  in  the  1,000  function 
point  size  range  indicate  that  the  CMM 
and  CMMI  are  somewhat  more  cumber¬ 
some  than  the  Agile  methods  and  there¬ 
fore  tend  to  have  somewhat  lower  pro¬ 
ductivity  rates  (see  Table  2).  That  is  not 
to  say  that  the  CMM/ CMMI  methods 
produce  bad  or  unacceptable  results,  but 
only  that  the  Agile  methods  appear  to 
generate  higher  productivity  levels. 

However,  if  the  size  plateau  for  com¬ 
parison  is  upped  to  10,000  function 
points  or  100,000  function  points,  the 
CMM/ CMMI  methods  would  pull  ahead. 
At  10,000  function  points  the  overall  staff 
would  be  larger  than  50,  while  for 
100,000  function  points  the  overall  staff 
would  approach  600  people  and  some  of 
the  Agile  principles  such  as  daily  team 
meetings  would  become  difficult  and  per¬ 
haps  impossible. 

Also,  applications  in  the  10,000  to 
100,000  function  point  size  range  tend  to 
have  thousands  or  even  millions  of  users. 
Therefore  it  is  no  longer  possible  to  have 
direct  daily  contact  with  users  since  their 
numbers  are  too  large. 

When  quality  is  considered,  the  Agile 
approach  of  having  frequent  daily  con¬ 
tacts  with  users,  daily  Scrum  sessions,  and 
writing  the  test  cases  before  the  code  has 
the  effect  of  lowering  defect  potentials.  The 
phrase  defect  potentials  refers  to  the  total 
number  of  defects  that  might  be  encoun¬ 
tered  in  requirements,  design,  coding, 
user  documents,  as  well  as  had fixes  or  sec¬ 
ondary  defects.  (The  current  U.S.  average 
would  be  about  5.0  defects  per  function 
point  for  defect  potentials.)  However, 
Agile  projects  usually  do  not  perform 
formal  design  and  code  inspections  so 
their  defect  removal  efficiency  is  some¬ 
what  below  the  CMM  example. 

The  defect  potentials  for  the  CMM 
version  are  better  than  U.S.  averages  but 
not  quite  equal  to  the  Agile  version  due 
to  the  more  detached  connections 
between  clients  and  developers. 

However,  the  CMM  and  CMMI 
methods  typically  do  utilize  formal  design 
and  code  inspections.  As  a  rule  of  thumb, 
formal  inspections  are  more  than  65  per¬ 
cent  efficient  in  finding  bugs  or  defects, 
which  is  about  twice  the  efficiency  of 
most  forms  of  testing,  many  of  which 
only  find  about  30  percent  of  the  bugs 
that  are  actually  present. 

As  of  2007  the  current  U.S.  average 
for  cumulative  defect  removal  efficiency 
is  only  about  85  percent,  so  both  the 
Agile  and  CMM  examples  are  better  than 
U.S.  norms.  The  CMM  method  is  some¬ 
what  higher  than  the  Agile  method  due  to 
formal  inspections,  testing  specialists,  and 


a  more  formal  quality  assurance 
approach. 

As  a  general  rule,  methods  developed 
to  support  large  software  applications 
such  as  the  CMM  and  CMMI  are  cum¬ 
bersome  when  applied  to  small  projects. 
They  can  be  tailored  or  subset  to  fit  small 
projects,  but  out  of  the  box  they  are  cum¬ 
bersome. 

On  the  other  hand,  methods  devel¬ 
oped  to  support  small  software  applica¬ 
tions  such  as  the  Agile  methods  become 
less  and  less  effective  as  application  sizes 
increase.  Here  too,  they  can  be  tailored  or 
modified  to  fit  larger  projects,  but  out  of 
the  box  they  are  not  effective. 

As  of  2007  both  the  Agile  and 
CMM/ CMMI  communities  are  working 
on  merging  the  more  useful  features  of 
both  sets  of  methods.  Agile  and  the 
CMM/CMMI  are  not  intrinsically 
opposed  to  one  another,  they  merely 
started  at  opposite  ends  of  the  size  spec¬ 
trum. 

Overall  Observations  on 

Software  Development 

With  more  than  13,000  projects  exam¬ 
ined,  it  can  be  stated  categorically  that 
there  is  no  single  method  of  develop¬ 
ment  that  is  universally  deployed  or  even 
universally  useful.  All  software  projects 
need  some  form  of  gathering  require¬ 
ments,  some  form  of  design,  some  kind 
of  development  or  coding,  and  some 
forms  of  defect  removal  such  as 
reviews,  inspections,  and  testing. 

In  the  author’s  studies  more  than  40 
methods  for  gathering  requirements, 
more  than  50  variations  in  handling  soft¬ 
ware  design,  more  than  700  program¬ 
ming  languages,  and  more  than  30  forms 
of  testing  were  noted  among  the  pro¬ 
jects  examined.  Scores  of  hybrid  meth¬ 
ods  have  also  been  noted. 

Both  the  Agile  methods  and  the 
CMM/ CMMI  methods  have  added 
value  to  the  software  industry.  But  care 
is  needed  to  be  sure  that  the  methods 
selected  are  in  fact  tailored  to  the  size 
range  of  the  applications  being  con- 
structed.^ 
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Is  CMMI  Useful  and  Usable  in  Small  Settings? 

One  Example 
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The  Software  Engineering  Directorate  (SED)  of  the  U.S.  Mrmj  Mviation  and  Missile  Kesearch,  Development  and 
Engineering  Center  (AMRDEC)  in  Huntsville,  Alabama,  acquires  software-intensive  systems  and  has  more  than  250 
small  companies  in  its  supply  chain.  In  order  to  determine  the  appropriateness  of  using  Capability  Maturif  Model  Integrated 
(CMMI®)  as  supplier  requirements,  members  of  AMRDEC  SED  teamed  with  the  Software  Engineering  Institute  (SEI) 
to  perform  a  technical  feasibility  study  in  2003-2004.  This  article  presents  the  motivation,  the  processes  used,  and  the  major 
results  of  the  CMMI  for  Small  Business  pilot  from  the  perspective  of  the  team  that  worked  on  the  pilot. 


WT  often  hear  that  CMMI®  was  At  built 
for  small  companies  so  it  will  not  work  for 
them^  or  some  variant  of  this  sentiment. 
Many  people  find  the  CMMI  book/ tech¬ 
nical  report  intimidating  to  think  about 
using  it.  Although  it  is  true  that  CMMI 
was  not  explicitly  built  for  small  compa¬ 
nies,  it  is  also  true  that  it  was  not  explic¬ 
itly  built  for  large  companies  [1].  The 
experience  we  obtained  from  the  CMMI 
for  Small  Business  pilot  indicates  that 
CMMI,  when  applied  in  a  way  that 
responds  to  the  business  realities  of  a 
small  business,  can  provide  small  compa¬ 
nies  with  utility. 

Small  Pilot  Company  Profiles 

Two  small  companies  from  Huntsville, 
Alabama  were  selected  to  participate  in 
the  pilot: 

•  Analytical  Sciences,  Incorporated 
(ASI)  specializes  in  management  and 
technical  services  with  a  focus  on  sys¬ 
tems  engineering/ program  manage¬ 
ment,  information  technology,  engi¬ 
neering  and  scientific  services,  and 
professional  and  organizational  devel¬ 
opment. 

•  Cirrus  Technologies,  Incorporated 

(CTI)  specializes  in  manufacturing 
and  support  services  with  a  focus  on 
logistics,  engineering,  manufacturing, 
test  and  evaluation,  information  tech¬ 
nology,  security,  and  intelligence. 

At  the  time  of  the  pilot,  each  company 
had  around  200  employees.  The  projects 
selected  for  the  pilot  ranged  in  size  from  a 
one-person  project  to  a  22-person  project. 
CMMI  vl.l  SE/SW  was  used  as  the  refer¬ 
ence  model  for  the  project. 

Key  Challenges  in  Process 
Improvement  for  Small 
Business 

We  saw  several  challenges  during  the 
adoption  pilot  in  Huntsville.  Some  were 
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challenges  that  we  had  hypothesized, 
some  were  new  insights.  Although  the 
pilot  was  not  designed  to  address  aU  of 
these  challenges,  we  list  them  here  in  the 
following  as  a  reference  to  underscore  that 
we  acknowledge  that  there  are  a  diverse 
set  of  challenges  for  CMMI  adoption  in  a 
small  setting: 

•  AffordabiHty  of  process  improvement 
is  a  major  challenge. 

**Ensure  that  senior 
management 
understands  how  to 
interpret  appraisal 
results,  both  in  terms  of 
what  they  are  likely  to 
mean  in  terms  of 
performance  and  how 
they  can  be  appropriately 
used  in  marketing 
concepts. 

•  Small  businesses  need  to  realize  payoff 
quickly. 

•  Small  businesses  do  not  have  staff 
dedicated  solely  to  process  improve¬ 
ment  implementation:  Customer 
requirements  take  priority  and  can 
cause  delays. 

•  There  is  minimal  structure  to  leverage 
from  in  a  small  business. 

•  The  customer  rules.  Many  small  organiza¬ 
tions  adopt/ adapt  their  business  prac¬ 
tices  directly  from  their  customers  or 
prime  contractor. 

•  If  a  quality  system  is  either  not  already 
in  place  or  is  not  well-functioning. 


process  definition  efforts  are  much 
more  challenging. 

•  CMMI  is  generally  perceived  as  intim¬ 
idating,  both  in  size  and  scope. 

Motivation  for  the  Pilot 

The  AMRDEC  SED  is  one  of  three  Life 
Cycle  Software  Engineering  Centers  in  the 
Army.  Established  in  1984,  the  SED  is  a 
recognized  leader  in  supporting  the  acqui¬ 
sition,  research,  development,  and  sustain¬ 
ment  of  some  of  the  nation’s  most 
sophisticated  weapon  systems.  The  mis¬ 
sion  of  the  SED  is  to  provide  mission  crit¬ 
ical  computer  resource  expertise  to  sup¬ 
port  weapon  systems  over  their  life  cycle. 
This  mission  is  carried  out  by  a  staff  of 
approximately  900  government  and  con¬ 
tractor  employees  housed  in  the  Army’s 
only  facility  designed  specifically  for  tacti¬ 
cal  battlefield  automated  systems  support. 

Like  many  federal  organizations,  the 
SED  relies  heavily  upon  a  contract  work¬ 
force  for  the  fulfillment  of  its  mission. 
The  two  primary  SED  contract  vehicles 
consist  of  many  companies  categorized  as 
small  businesses.  Currently,  more  than  75 
percent  of  the  companies  contracted  for 
engineering  services  with  the  SED  are 
small  businesses.  Since  these  companies 
are  increasingly  involved  in  the  develop¬ 
ment  of  significant  components  for  soft¬ 
ware -intensive  systems,  their  usage  of  reli¬ 
able  engineering  and  management  prac¬ 
tices  has  become  increasingly  critical  to 
the  delivery  of  quality  products  for  the 
Department  of  Defense  (DoD)  warfight¬ 
er. 

Pilot  Process  Overview 

The  CMMI  for  Small  Business  pilot  start¬ 
ed  in  July  2003  and  culminated  in  May 
2004  with  a  Standard  CMMI-based 
Appraisal  Method  for  Process  Improve¬ 
ment  vl.l  (SCAMPER  Class  A  appraisal 
of  each  of  the  two  pilot  companies.  The 
overall  process  is  summarized  in  Figure  1. 
Gaps  between  the  organizations’  internal 
processes  and  CMMI  were  identified  by 
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engaging  in  a  collaborative  session 
between  the  pilot  team  consultants  and 
the  practitioners  from  the  pilot  companies 
that  was  similar  to  a  SCAMPI  C  appraisal. 
Based  on  the  results  of  this  analysis,  the 
pilot  companies  developed  and  imple¬ 
mented  an  action  plan  and  updated  exist¬ 
ing  processes  to  close  the  gaps  found. 
Where  necessary,  the  pilot  companies  also 
developed  new  processes.  Though  we  ini¬ 
tially  did  not  intend  to  perform  SCAMPI 
A  appraisals,  the  progress  made  by  both 
companies  was  such  that  in  January  of 
2004  we  defined  appraisal  scopes  in  con¬ 
junction  with  the  pilot  companies,  and  in 
May  2004  we  performed  SCAMPI  vl.l  A 
appraisals  using  the  continuous  represen¬ 
tation  of  CMMI-SE/SW  vl.l  at  both  sites 

[2] .  Both  companies  achieved  their  target 
level  profiles,  as  foUows: 

•  ASI:  Capability  Level  2  for  project 
planning,  requirements  management, 
and  measurement  and  analysis,  and 
Capability  Level  3  for  organization 
process  focus  and  organizational  train- 
ing. 

•  CTI:  Capability  Level  1  for  project 
planning,  requirements  management, 
and  project  monitoring  and  control 
(given  some  of  the  other  business 
challenges  that  CTI  was  facing  at  the 
time  of  the  pilot,  establishing  Level  1 
processes  in  these  areas  was  a  signifi¬ 
cant  achievement). 

Lessons  Learned  From  the 
Pilot 

There  are  several  competencies  in  process 
improvement  that  provide  a  useful  frame¬ 
work  for  looking  at  lessons  learned  from 
the  pilot  study.  Four  of  these  are  included 
here  as  a  way  to  organize  lessons  learned 

[3] : 

•  Establishing  and  sustaining  sponsor¬ 
ship. 

•  Developing  infrastructure/ defining 
processes. 

•  Deploying  new  processes  into  the 
intended  use  environment. 

•  Managing  an  appraisal  life  cycle. 


We  have  included  an  additional  catego¬ 
ry  of  lessons  learned  in  this  section: 
lessons  about  the  CMMI  model  itself 
Those  readers  who  are  experienced  in 
process  improvement  consulting  in  a  vari¬ 
ety  of  settings  may  recognize  our  primary 
competencies  as  categories  that  also  apply 
to  larger  organizations.  However,  the  par¬ 
ticular  lessons  that  have  been  included 
here  are  those  that  we  believe  are  either 
unique  to  the  small  settings  environment 
or  are  particularly  important  for  a  small 
company  to  be  successful  in  their 
improvement  efforts. 

Establishing  and  Sustaining 
Sponsorship 

Obtaining  and  sustaining  the  executive 
sponsorship  necessary  to  make  applying 
resources  to  process  improvement  activi¬ 
ties  feasible 

•  Lesson  1:  Focus  CMMI  implementa¬ 
tion  in  areas  where  the  connection 
between  the  model’s  content  and  the 
Chief  Executive  Officer’s  (CEO)  busi¬ 
ness  goals  are  clearest. 

In  a  small  company,  sponsorship 
often  means  getting  the  attention  of 
the  owner  and/ or  CEO  of  the  compa¬ 
ny.  In  this  setting,  the  focus  of  the 
CEO  is  often  on  a  combination  of 
cash  flow  management  and  develop¬ 
ment  of  the  growth  of  the  company. 
This  implies  that  any  process  improve¬ 
ment  efforts  that  are  presented  must 
be  aligned  with  the  particular  financial 
environment  and  growth  goals  of  the 
company. 

•  Lesson  2:  Even  if  you  do  not  have 
strong  quantitative  results  right  away, 
make  sure  that  the  senior  management 
gets  periodic  progress  reports  that 
include  the  qualitative  benefits  of  the 
improvement  effort. 

•  Lesson  3:  Ensure  that  senior  manage¬ 
ment  understands  how  to  interpret 
appraisal  results,  both  in  terms  of  what 
they  are  likely  to  mean  in  terms  of  per¬ 
formance  and  how  they  can  be  appro¬ 
priately  used  in  marketing  contexts. 


Developing  Infrastructure/ Defining 
Processes 

Providing  enabling  infrastructure  to 
make  definition  and  use  of  new  processes 
effective 

Examples  of  activities  that  fit  in  this  cate¬ 
gory  include  the  following: 

°  Establishing/managing  a  pro¬ 
cess  asset  library. 

°  Establishing/managing  a  mea¬ 
surement  repository. 

°  Establishing/ maintaining  stan¬ 
dards,  approaches,  and  accept¬ 
ed  methods  for  writing  process 
guidance. 

°  Establishing/managing  the  or¬ 
ganization’s  curriculum  for  pro¬ 
cess  improvement. 

°  Establishing  points  of  contact 
or  specific  groups  (e.g.,  an  engi¬ 
neering  process  group  [EPG]) 
for  various  aspects  of  the 
improvement. 

•  Lesson  4:  Even  though  a  formal  EPG 
may  be  infeasible  for  small  companies, 
some  focal  point  for  coordination  is 
particularly  needed  to  coordinate 
infrastructure  development  and  sus¬ 
tainment. 

•  Lesson  5:  When  a  well- functioning 
quality  management  system  is  already 
in  place  (e.g.,  based  on  International 
Organization  for  Standardization 
PSO]  9001),  take  advantage  of  it!  The 
existence  of  a  well- functioning  ISO 
9001 -based  quality  management  sys¬ 
tem  provided  a  bootstrap  for  process 
guidance  standards  and  several  other 
elements  of  process  improvement 
infrastructure.  On  the  other  hand,  if 
there  had  been  no  quality  system 
already  in  place,  some  time  would  have 
been  needed  to  establish  and  set  up 
procedures  for  using  some  kind  of 
mechanism  for  storing,  controlling, 
and  distributing  process  assets  created 
as  part  of  the  improvement  effort. 

•  Lesson  6:  The  tools  and  practices  of 
the  accounting  system  have  a  great 
influence  on  what  is  considered  doable 
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in  terms  of  collecting  and  using  mea¬ 
surements.  A  small  company  typically 
does  not  have  the  resources  available 
to  create  a  parallel  metrics  collection 
system  from  their  mainstream 
accounting  system,  so,  at  least  at  the 
beginning,  what  is  considered  feasible 
in  terms  of  measurement  is  con¬ 
strained  by  what  can  be  collected/ 
aggregated  by  the  tools  in  use. 

Deploying  New  Processes  Into  the 
Intended  Use  Environment 
Ensuring  that  the  new  CMMI-informed 
processes  are  available  to  all  relevant 
users  and  that  their  successful  adoption  is 
associated  with  appropriate  training  and 
job  aids.  This  is  where  much  of  what  we 
traditionally  call  organization  change 
management  occurs 

•  Lesson  7:  Simple  CMMI-based 
improvements  can  have  a  significant 
impact  in  small  organizations. 

In  one  case,  just  adding  meeting 
minutes  for  the  weekly  meeting  and 
publicizing  them  to  the  customer  and 
project  participants  (not  more  than 
five  people  total)  contributed  to  more 
efficient  monitoring  of  the  project  and 
improved  communication  between  the 
customer  and  the  project  team.  It 
sounds  simple,  and  it  is:  The  model 
provided  an  incentive  to  try  something 
so  there  would  be  records  of  deci¬ 
sions/status  progress.  However,  the 
effect  was  much  greater  than  the  pro¬ 
ject  participants  anticipated,  both  in 
terms  of  scope  of  effect  and  magni¬ 
tude  —  the  change  not  only  provided 
an  effective  tool  for  monitoring  but  it 
also  resulted  in  improved  communica¬ 
tion  with  the  customer,  which  greatly 
improved  the  performance  of  the  pro¬ 
ject  as  a  whole. 

Seeing  unanticipated  benefits 
from  small  changes  was  a  great  moti¬ 
vator  for  continuing  on  the  path  of 
improvement  and  being  willing,  a  little 
later  in  the  process,  to  try  larger 
changes.  In  small  companies,  the 
effects  of  small  changes  can  often  be 
seen  much  more  quickly  and  the  dis¬ 
persal  of  knowledge  throughout  the 
company  about  the  effects  of  a  change 
is  also  faster. 

Managing  an  Appraisal  Life  Cycle 
Selecting  a  method  of  measuring 
progress  against  a  model  (i.e.,  appraisal 
method)  and  then  planning  and  executing 
the  tasks  associated  with  the  selected 
method 

•  Lesson  8:  Use  a  focused,  collaborative 
appraisal  method  (e.g.,  SCAMPI  B  or 


C)  for  the  initial  gap  analysis.  Great 
benefit  is  realized  by  using  this  session 
as  an  opportunity  to  interpret  the 
model  and  gain  a  better  understanding 
of  how  it  applies  to  the  organization. 

•  Lesson  9:  Ensure  someone  in  the 
organization  has  a  good  understanding 
of  Appraisal  Requirements  for  CMMI 
Class  A,  B,  and  C  appraisal  methods 
and  set  expectations  [4].  This  greatly 
increases  the  potential  for  achieving 
the  appraisal  objectives  defined  by  the 
appraisal  sponsor. 

•  Lesson  10:  Collect  evidence  that  will 
be  useful  in  the  appraisal  as  you  go 
using  automation  support  as  much  as 
possible.  Interact  with  the  lead 
appraiser  during  evidence  collection 
and  mapping  to  CMMI  practices  to 
ensure  that  a  complete,  well-organized 
set  of  evidence  is  available  for  the 
appraisal.  This  does  not  need  to  be 
days  and  days  of  billable  interaction.  It 
may  just  take  the  form  of  e-mailing 
templates  for  evidence  collection  to 
get  an  idea  of  how  they  fit  with  the 
lead  appraiser’s  expectations. 

Although  this  is  one  of  the 
lessons  that  is  also  equally  applicable  in 
a  larger  setting,  the  effects  if  this  is 
NOT  done  are  much  greater  in  a  small 
setting  in  terms  of  the  percentage  of 
staff  time  that  has  to  be  used  to 
rework  material  that  has  been  prepared 
for  the  appraisal. 

•  Lesson  11:  Introduce  generic  prac¬ 
tices  once  specific  practices  are  clearly 
understood  but  prior  to  the  definition 
and  documentation  of  processes. 
Misinterpretation  of  generic  practices  is  a 
major  cause  for  appraisal  failures.  This  is 
an  area  where  investing  in  a  small 
amount  of  external  consulting  could 
pay  big  benefits.  In  the  case  of  the 
pilot  projects,  we  held  a  generic  prac¬ 
tices  workshop  to  help  the  pilot  par¬ 
ticipants  get  a  better  understanding  of 
the  linkages  between  generic  practices 
and  the  process  areas  they  were  work¬ 
ing  with. 

•  Lesson  12:  Quick  looks  (e.g.,  SCAMPI 
B  and  SCAMPI  C)  significantly 
improve  the  chances  for  achieving  the 
objectives  of  a  SCAMPI  A. 

CMM!  Model 

•  Lesson  13:  Overall,  we  saw  that  judi¬ 
cious  use  of  the  elements  of  CMMI 
that  relate  to  the  business  context 
provided  a  set  of  useful  practices 
from  which  small  businesses  can  ben¬ 
efit,  though  not  always  in  predicted 
ways. 

•  Lesson  14:  Using  the  continuous  rep¬ 


resentation  of  the  model  allowed  the 
pilots  to  focus  on  improvements  that 
they  perceived  as  having  the  highest 
payoff  for  the  company. 

•  Lesson  15:  Changing  the  practices  in 
the  model  is  not  necessary  in  most 
cases;  finding  alternative  practices  is 
often  relevant.  In  addition,  work  prod¬ 
ucts  generated  as  a  result  of  practice 
implementation  rarely  match  one-to- 
one  to  what  is  suggested  in  the  model. 

•  Lesson  16:  Smallness  was  not  as  much 
of  an  issue  for  model  interpretation 
as  the  focus  of  the  business. 
Although  both  organizations  had  a 
more  traditional  product  develop¬ 
ment  project  included  in  their  pilot, 
they  also  had  more  pure  service  deliv¬ 
ery  contexts  (give  me  a  team  of  N 
people  who  can  do  X  for  25  hours  per 
month  for  the  next  six  months)  that 
they  wanted  to  explore  because  ser¬ 
vice  delivery  is  the  heart  of  their  busi¬ 
ness.  Sometimes  those  services  are 
delivered  in  the  context  of  a  project, 
but  they  often  are  not.  The  model  was 
more  difficult  to  interpret  in  areas  of 
the  pilot  involved  in  service  delivery 
than  in  the  small  product  develop¬ 
ment  projects.  The  SEI  is  involved  in 
an  effort  led  by  Northrop  Grumman 
to  develop  a  CMMI  for  Services 
(SVC)  constellation  that  may  prove 
more  useful  in  this  context. 
Information  on  CMMI-SVC  can  be 
found  on  the  CMMI  Web  site  at 
<www.sei.cmu.edu/ cmmi>. 

A  Toolkit  to  Help  You  Start 
Your  Own  CMMI-Based 
Improvement  Effort 

As  a  major  product  of  the  pilots,  the  team 
produced  a  Web-based  toolkit  that  pro¬ 
vides  details  on  the  processes  and  assets 
used  in  the  pilot.  (The  draft  of  the  toolkit 
can  be  found  at  <www.sei.cmu.edu/ 
cmmi/publications /toolkit/ >.  It  is  a  draft 
that  was  not  fuUy  completed  due  to  bud¬ 
get  constraints.  It  may  get  incorporated 
into  the  Implementing  Process  Improve¬ 
ment  in  Small  Settings  pPSS]  Field  Guide, 
in  which  case  it  would  be  updated.)  In 
addition  to  process  descriptions,  it  pro¬ 
vides  copies  of  the  actual  presentations, 
templates,  and  other  documents  used  to 
support  the  pilot.  It  should  be  treated  as 
an  anecdotal  set  of  assets  that  might  be 
useful  in  supporting  a  model-based 
improvement  effort,  rather  than  a  canoni¬ 
cal  set  that  defines  what  should  be  used. 
Having  said  that,  we  believe  that  the  toolk¬ 
it  can  help  people  working  on  improve¬ 
ment  in  the  following  small  settings: 
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•  Focus  their  improvement  efforts. 

•  Figure  out  how  and  where  to  get  start¬ 
ed. 

•  Tie  their  improvements  to  business 
goals. 

•  Educate  their  staff  in  areas  where  they 
may  need  to  improve  their  knowledge 
and  skills. 

•  Realize  payoffs  (mostly  qualitative) 
early  in  the  improvement  effort. 

•  Improve  their  ability  to  prepare  for 
appraisals. 

The  feedback  to  date  that  we  have 
received  on  the  toolkit  has  been  very  pos¬ 
itive  and  fairly  broad  in  terms  of  global 
access  (people  from  Argentina,  Israel, 
United  Kingdom,  Mexico,  and  Chile,  as 
well  as  the  U.S.  and  Canada  have  accessed 
the  toolkit). 

In  thinking  about  using  the  toolkit,  we 
have  a  few  recommendations  for  those 
who  are  working  in  small  settings  current¬ 
ly  and  are  planning  to  use  it  to  support 
your  improvement  effort: 

•  Think  of  this  as  one  resource  to  help 
you,  but  not  the  only  one.  Every 
month  there  are  new  publications 
related  to  CMMI;  some  of  them  are 
likely  to  offer  different  insights  than 
the  toolkit  but  they  may  be  valuable  to 
you. 

•  Be  sure  to  read  the  Whafs  Missing  sec¬ 
tion  of  the  toolkit  to  see  if  any  of  the 
things  we  talk  about  in  that  section 
apply  to  you.  If  they  do,  then  you 
know  you  will  need  resources  beyond 
what  we  used  to  get  you  started  and  be 
successful. 

•  For  those  of  you  who  are  in  the  DoD 
supply  chain,  think  about  getting  men¬ 
toring  from  the  larger  companies  that 
work  with  you  and  have  ongoing 
improvement  efforts;  they  should  have 
a  vested  interest  in  your  success. 

•  Keep  up  with  the  assets  in  the  CMMI 
adoption  area  (<www.sei.cmu.edu/ 
cmmi/adoption>);  that  is  where  you 
will  see  emerging  work  on  CMMI  in 
small  settings  in  particular  and  other 
resources  that  may  be  of  value  to  you. 

•  Explore  at  a  reasonable  pace.  Unless 
you  have  some  business  investment 
riding  on  achievement  of  some  partic¬ 
ular  status  related  to  CMMI,  do  not  try 
to  do  too  much  at  once  until  you  have 
established  what  benefits  you  can 
accomplish  in  your  own  environment. 

Next  Steps 

SED^s  Plans  for  Follow-On  Activities 

The  CMMI  small  business  pilot  has  been 
one  of  the  most  beneficial  endeavors  of 
the  SED/SEI  strategic  partnership.  We 


are  pleased  that  the  AMRDEC  SED- 
sponsored  pilot  provided  the  stimulus  for 
the  establishment  of  the  IPSS  project  at 
the  SEI.  One  of  the  early  events  of  this 
project  was  an  International  Research 
Workshop  in  this  topic  area  that  was  held 
at  the  SEI  in  October  2005  and  resulted 
in  an  SEI  Technical  Report  summarizing 
the  workshop  and  containing  the  papers 
submitted  to  the  workshop.  This  report  is 
available  for  download  in  the  publications 
section  of  the  SEI  Web  site  [5] . 

As  the  SED/SEI  partnership  contin¬ 
ues,  we  will  start  to  gain  insight  into  the 
use  of  some  other  SEI  technologies  with¬ 
in  the  SED  setting.  These  include  the 
insertion  of  Personal  Software  Process^^/ 
Team  Software  Process^^  technology  in  an 
Army  pilot  program  to  provide  the  acqui- 

'Tor  those  of  you  who 
are  in  the  DoD  supply 
chain,  think  about  getting 
mentoring  from  the 
larger  companies  that 
work  with  you  and  have 
ongoing  improvement 
efforts;  they  should  have 
a  vested  interest  in 
your  success/* 


sition  organization  with  greater  insight 
into  development  metrics.  Additionally, 
the  SED/SEI  partnership  serves  an  inte¬ 
gral  role  in  providing  acquisition  process 
improvement  support  to  many  of  our 
local  Army  program  managers. 

SEPs  Plans  for  Supporting  CMMI  for 
Small  Settings 

The  pilot  project  in  Huntsville,  Alabama 
emphasized  to  the  SEI  the  need  for 
appropriate  guidance  materials  for  using 
CMMI  in  small  settings.  In  response,  the 
SEI  has  chartered  the  IPSS  project  within 
the  International  Process  Research 
Consortium  initiative.  Seed  funding  result¬ 
ed  in  the  International  Research  Work¬ 
shop  mentioned  earlier,  and  initial  spon¬ 
sors  are  supporting  the  prototyping  of  an 
IPSS  Field  Guide  that  reflects  many  of  the 
lessons  cited  here.  Contact  Caroline 
Graettinger,  the  IPSS  project  manager,  for 
details,  at  <cpg@sei.cmu.edu>. 


Conclusion 

We  hope  you  wiU  find  this  information 
beneficial  as  you  embark  on  your  own 
improvement  journey  and  you  will 
become  a  member  of  the  burgeoning 
community  of  practice  for  CMMI  in  small 
settings.  Stay  tuned  with  ongoing  SEI 
work  in  small  settings  at  <www.sei. 
cmu.edu/iprc/ipss.html>.  This  endeavor 
is  discussed  more  on  page  27. ♦ 
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Why  Do  I  Need  All  That  Process? 

Tm  Only  a  Small  Project^ 

Mark  Brodnik,  Robyn  Blouse,  and  Terry  Leip 

Intel  Corporation 

A.t  Intel’s  Information  Technology  (IT)  department,  we  developed  extensive  processes  for  our  projects.  While  the  large  projects 
get  the  glory,  the  majority  of  our  projects  are  less  than  six  months  long,  have  small  teams,  limited  scope,  and  low  risk.  We 
found  that  we  have  a  variety  of  project  si^es  hut  a  single  set  of  processes  originally  built  for  larger  projects.  So  how  did  we 
fix  that  issue? 


The  Intel  IT  department,  like  many 
other  organizations,  was  seeing  issues 
with  completing  software  projects  on  time 
and  satisfying  our  customers.  Using  the 
Software  Engineering  Institute’s 
Capability  Maturity  Model  Integration 
(CMMP^  as  a  basis  to  tackle  those  issues, 
management  challenged  us  with  the  goal 
of  being  at  Maturity  Level  3  in  1 8  months. 
While  we  acknowledged  that  this  goal  was 
overly  ambitious,  we  still  took  a  shot  at 
accomplishing  the  objective.  A  small  team 
using  the  CMMI  as  their  guide  built  large 
and  formal  processes  and  mandated  their 
use.  The  inevitable  result  was  that  this 
approach  was  not  well  received  or  imple¬ 
mented  by  the  project  teams,  especially 
those  at  the  smaller  end  of  the  scale.  The 
IT  organization  also  realized  that  there 
was  no  business  justification  for  a  CMMI 
benchmark.  To  address  this  issue,  we 
launched  a  project  to  streamline  the 
processes;  we  took  a  different  look  at  the 
CMMI,  listened  to  the  stakeholders,  and 
focused  on  our  business  objectives.  The 
end  result  was  a  set  of  processes  70  per¬ 
cent  shorter  and  much  less  prescriptive. 

Accelerated  Process 
Improvement  (API) 

Based  on  stakeholder  feedback,  audit 
results,  and  process  coach  input,  we  iden¬ 
tified  a  number  of  the  work  processes 
that  were  complex  and  difficult  to  use. 
The  organization  gathered  35  representa¬ 
tives  including  key  stakeholders,  engi¬ 
neers,  process  coaches,  and  auditors  for 
an  intense  three-and-a-half  day  face-to- 
face  meeting.  In  preparation  for  the  ses¬ 
sion  we  mapped  business  and  model 
requirements  to  our  existing  processes  in 
order  to  determine  what  portions  were 
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eligible  for  simplification.  In  addition,  we 
created  a  priority  list  of  feedback  from 
audit,  coaching,  and  pending  process 
improvement  requests  to  determine  top 
pain  points  and  focus  areas.  Since  we  had 
limited  face-to-face  time  to  complete  the 
work,  we  were  willing  to  accept  less  than 
perfect  results  and  less  adherence  to  the 
model  to  streamline  the  business  process. 

A  key  component  of  the  successful 
collaboration  was  sequestering  the  team 
at  an  off-site  facility  to  maximize  focus 
and  support  a  collaborative  environment. 
The  agenda  for  this  event  was  driven  by 
the  stakeholders  rather  than  the  process 
engineers.  The  stakeholders  drove  the 
vast  majority  of  the  changes  with  a  pri¬ 
mary  focus  on  accomplishing  business 
results  rather  than  textbook  model  com¬ 
pliance.  The  team  reviewed  each  work 
process  against  the  business  require¬ 
ments  and  the  CMMI  model  to  identify 
portions  of  the  process  that  could  be 
considered  for  deletion  or  simplification. 
The  group  was  broken  down  in  to  three 
sub-teams  each  addressing  a  different 
process  area.  Daily  report  out  sessions 
and  synchronization  sessions  helped 
ensure  consistency  between  the  teams. 
Our  CMMI  model  expert  floated 
between  the  teams  to  answer  model  relat¬ 
ed  questions. 

One  of  the  themes  we  noticed  was 
that  the  initial  process  development  team 
had  implemented  many  of  the  sub-prac¬ 
tices  in  the  CMMI  model  believing  they 
were  a  required  model  component.  For 
example,  based  on  sub-practices  from  the 
configuration  management  process  area, 
we  required  each  document  to  have 
unique  identifier  labels  both  internal  to 
the  document  and  in  the  file  name  —  a 
convention  that  was  unnecessarily  com¬ 
plex  and  added  little  value  for  our  typical 
8-to-12  person  project  teams.  We  also 
defined  and  mandated  specific  elicitation 
techniques  to  meet  the  intent  of  the  sub¬ 
practices  in  the  requirements  develop¬ 
ment  process  area;  these  techniques  were 
overkill  for  our  small  project  teams  which 


worked  closely  with  their  customers  and 
stakeholders.  Finally,  we  eliminated  the 
requirement  for  a  separate  formal  com¬ 
mitment  for  resources  by  combining  this 
with  one  of  the  key  milestone  decisions. 
This  came  about  by  an  incorrect  interpre¬ 
tation  of  the  sub-practices  in  the  project 
monitoring  and  control  process  area. 

Figure  1  (see  page  20)  shows  the  old 
processes  for  scope,  schedule,  and 
resource  management.  These  separate 
processes  totaled  29  pages  and  17 
process  steps.  The  new  process  was  able 
to  combine  scope,  schedule,  and  resource 
management  into  one  process  document 
that  was  five  pages  long  with  eight 
process  steps.  This  was  accomplished  by 
simplifying  the  wording,  combining  or 
eliminating  process  steps,  and  removing 
instructional  text  which  we  thought 
would  substitute  for  skills  expertise  from 
the  process  documents. 

Overall,  we  achieved  a  70  percent 
reduction  in  document  length,  with  one 
process  shrinking  to  just  17  percent  of  its 
original  size.  Detail  of  other  process  and 
template  reductions  can  be  seen  in  Table  1 
(see  page  20).  The  original  and  combined 
processes  can  be  accessed  via  the  online 
version  of  this  article. 

When  we  evaluated  the  outcome  of 
the  effort,  one  of  our  project  managers 
said  the  following: 

The  API  face-to-face  session  was 
great  for  analyzing  the  weak  spots 
(in  the  documentation  and  in  the 
processes),  as  well  as  acknowledg¬ 
ing  the  flaws  of  the  previous 
release.  The  teams  addressed  the 
issues  with  great  teamwork  that 
allowed  for  development  of  some 
creative  and  tangible  action  plans. 

In  the  end,  we  reduced  the  workload  of 
our  processes  and  improved  documenta¬ 
tion  so  that  it  would  more  effectively  deliv¬ 
er  key  information.  The  result  of  process 
simplification  was  the  ability  of  smaller 
projects,  originally  deemed  out  of  scope. 
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Original  Flows  for  Scope,  Schedule,  and  Resource  Processes 


Figure  1 :  Example  Simplification  and  Eduction 


to  adopt  the  processes.  Our  number  of  eli¬ 
gible  projects  went  from  36  percent  to  50 
percent.  Our  process  coaches  could  now 
support  an  average  of  18  projects  per 
coach,  up  from  14  projects  per  coach 
before  the  simplification.  In  addition,  the 
training  course  delivery  was  streamlined  to 
half  of  their  original  duration. 

What  About  Very  Small 
Projects? 

As  senior  management  started  seeing  the 
benefits  of  standard  processes,  they  made 
the  decision  to  require  all  projects  greater 
than  eight  weeks  in  duration  to  adopt  the 
processes.  As  the  adoption  rates  for  these 
projects  came  closer  to  100  percent,  we 
began  to  look  for  ways  to  help  the  pro¬ 
jects  originally  deemed  too  small  for  our 
processes.  These  very  small  projects,  less 
than  eight  weeks  in  duration,  were  con¬ 
sidered  low  risk  and  therefore  were  not 
required  to  follow  any  formal  processes 
or  standards,  although  some  did  have 


their  own  self-imposed  methods  of  run¬ 
ning  their  projects.  A  result  of  this  was 
that  management  did  not  have  sufficient 
visibility  into  these  projects  and  some  of 
the  larger  projects  would  attempt  to 
chunk  their  work  into  less  than  eight 
weeks  so  they  could  avoid  the  full 
process  and  external  visibility  into  their 
work. 

In  our  discussions  with  project  man¬ 
agers,  we  also  realized  that  the  eight-week 
rule  was  too  limiting,  since  many  projects 
were  only  slightly  longer  (8  to  12  weeks), 
but  had  just  two  or  three  fractional 
resources.  These  project  managers  felt 
that  the  overhead  of  the  processes  were 
still  too  large  compared  to  the  amount  of 
overall  effort  of  the  project,  and  they 
wanted  to  also  take  advantage  of  a  small¬ 
er  set  of  processes.  They  argued  that  they 
spent  most  of  their  time  filling  out  forms 
or  compliance  checklists  rather  than 
focusing  on  the  task  at  hand.  We  also  dis¬ 
covered  through  audits  that  project  man¬ 
agers  in  this  class  of  projects  would  often 


run  their  projects  as  usual  and  then  go 
back  after  the  fact  and  create  the  required 
process  collateral.  Clearly,  there  was  need 
to  provide  processes  that  addressed  very 
small  projects. 

Proof  of  Concept  (PoC) 

Before  we  deployed  a  new  set  of 
processes  across  all  of  IT,  we  decided 
the  best  approach  was  to  work  with  a 
limited  set  of  these  very  small  projects 
through  a  POC  in  one  or  two  IT  groups. 
We  would  then  analyze  the  data  to  deter¬ 
mine  if  an  IT-wide  deployment  was  war¬ 
ranted  or  if  we  needed  additional  pilots. 
Finally  we  planned  to  take  a  thorough 
look  at  our  entire  process  library  to 
implement  a  more  robust  project  charac¬ 
terization  approach  that  right-sizes  the 
processes  and  tools  used  by  our  project 
teams. 

The  first  area  we  looked  at  was  how 
our  process  deliverables  were  organized. 
Our  projects’  process  deliverables  are 
defined  at  two  main  levels.  The  first  level 
is  what  a  project  needs  to  review  at  a 
milestone  decision  meeting  with  man¬ 
agement  (ex.  requirements,  design,  and 
schedule).  The  second  level  is  deliver¬ 
ables  that  are  typically  produced  as  a 
result  of  project  work  (e.g.,  various  sup¬ 
porting  plans,  internal  compliance 
scorecards,  review  and  approval  of 
records,  change  records).  In  order  to 
address  the  issue  of  excessive  process 
overhead  for  small  projects,  the 


Table  1 :  Old  Versus  New  Document  Sfes 


Document 

Old  Size 

New  Size 

High-Level  Requirements  Process 

9  pages  -  7  steps 

4  pages  -  5  steps 

Detailed  Requirements  Process 

1 1  pages  -  6  steps 

4  pages  -  5  steps 

High-Level  Requirements  Template 

10  pages 

2  pages 

Detailed  Requirements  Template 

14  pages 

6  pages 

Scope/Schedule/Resource  Process 

29  pages  -  17  steps 

5  pages  -  8  steps 

Project  Plan  Plus  Supporting  Documents 

4  documents 

1  spreadsheet 

Baseline  Change  Control  Process 

6  pages  -  13  steps 

5  pages  -  9  steps 
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approach  to  the  PoC  was  to  determine 
the  major  process  deliverables  that  a 
project  should  produce  and  require  only 
those  deliverables.  To  speed  up  the  PoC, 
we  started  with  the  first-level  deliver¬ 
ables  and  decided  to  leave  the  creation 
of  second  level  process  deliverables  up 
to  the  discretion  of  the  project  manager. 
Ultimately,  the  goal  was  to  have  project 
managers  focus  more  on  the  product 
deliverables  and  less  on  the  process 
deliverables. 

The  next  item  we  looked  at  was  com¬ 
bining  the  templates  for  the  various 
process  deliverables  into  a  single,  simpli¬ 
fied  template  that  contains  the  basic 
information  the  projects  are  required  to 
complete.  This  allowed  decision  makers 
to  review  this  document  at  milestone 
decisions  rather  than  requiring  the  pro¬ 
ject  manager  to  duplicate  information  in 
a  separate,  formal  presentation. 

The  last  major  challenge  we  wanted 
to  work  through  in  the  PoC  was  the  cut¬ 
off  point  between  these  very  small  pro¬ 
jects  and  our  larger  projects.  In  the  end, 
we  decided  that  a  more  pragmatic 
approach  was  to  look  at  several  attribut¬ 
es  to  consider  for  the  cutoff  such  as 
team  size  (one  to  three  team  members), 
overall  effort  (around  500  hours),  and 
project  risk  (no  financial  systems  or  soft¬ 
ware  exchange  impact)  to  provide  a 
more  balanced  set  of  classifications. 
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Results 

While  the  model  is  not  our  primary  focus, 
we  are  stiU  measuring  ourselves  against  its 
requirements  to  make  sure  we  have  not 
missed  a  best  practice.  Through  a  series  of 
Standard  CMMI  Assessment  Method  for 
Process  Improvement^  (SCAMPI)  Cs,  Bs, 
and  As  performed  over  the  past  three 
years  we  are  seeing  increased  compliance 
with  the  model.  We  have  been  measuring 
progress  toward  Capability  Level  3  in 
Configuration  Management,  Project 
Planning,  Project  Management  and 
Control,  Requirements  Management,  and 
Requirements  Development,  and  we  are 
also  beginning  to  measure  compliance  for 
Measurement  and  Analysis,  Process  and 
Product  Quality  Assurance,  and  Verifica¬ 
tion  and  Validation. 

From  a  business  perspective,  we  are 
seeing  positive  results  in  shortening  the 
overall  duration  of  our  projects  and 
improving  our  ability  to  meet  our  com¬ 
mitted  delivery  dates.  This  is  done  by 
paying  particular  attention  to  the  planned 
versus  actual  time  in  each  individual 
phase. 

In  addition,  our  quality  assurance 
audit  team  discovered  that  process  com¬ 
pliance  had  improved  as  a  result  of  the 
simplification.  Since  the  processes  had 
changed  so  dramatically,  it  was  not  possi¬ 
ble  to  do  an  apples-to-apples  comparison 
of  the  audit  results  prior  to  simplifica¬ 


tion,  but  both  auditors  and  project  team 
feedback  indicated  that  compliance  was 
better  and  easier  to  achieve. 

Summary 

In  our  process  engineering  journey  we 
learned  the  following  things: 

•  First  and  foremost,  we  need  to  get  our 
stakeholders  involved  in  every  step  of 
the  development  process.  This  ensures 
quicker  acceptance  by  project  teams 
and  uncovers  usability  problems  earli¬ 
er  in  the  process. 

•  Second,  it  is  essential  to  have  a  busi¬ 
ness  focus  rather  than  a  model  focus. 
It  is  easy  to  lose  sight  of  business 
objectives  when  trying  to  comply  with 
CMMI  requirements. 

•  Third,  we  should  start  by  building 
processes  as  small  as  possible  and  add 
to  them  only  as  needed.  We  learned 
that  by  building  large  processes  first, 
the  true  cost  extends  beyond  the 
development  and  requires  significant 
effort  to  support  and  maintain. 

•  Finally,  time  is  the  most  precious  com¬ 
modity  in  process  improvement. 
Getting  a  quick  win  that  the  organiza¬ 
tion  would  accept  is  more  important 
than  perfection. 

In  the  end,  it  is  aU  about  being  both 
efficient  and  effective.^ 
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Small  Project  Survival  Among  the 
CM  Ml  Level  5  Big  Processes 


Alan  C.  Jost 
Raytheon 

In  the  past  several  years,  many  engineering  firms  have  stepped  up  and  achieved  Capability  Maturity  Model  Integrated 
(CMMI)  Level  5  [1].  Currently,  13.9  percent  ofi  the  2,140  appraised  organi^tions  have  achieved  CMMl  Level  5  using 
the  Standard  CMMl  Appraisal  Method  fior  Process  Improvement  (SCAMPI).  To  achieve  this  maturity  level,  the  organi- 
^tion  must  implement  a  rigorous  process  defiinition  to  address  the  CMMl  model  through  maturity  Level  5.  Once  defiined, 
the  organi^tional  process  must  be  implemented  throughout  the  organi^tion  even  by  small  projects.  I  will  examine  how,  in 
my  business  unit,  we  have  successfiully  achieved  CMMl  Level  5  across  fiive  geographically  dispersed  business  units  and  how 
small  projects  have  survived  within  our  signijicantly  large  process  dejinition. 


The  Raytheon  Network  Centric  Systems 
(RNCS)  has  five  major  engineering 
centers  across  the  country  An  interesting 
footnote  is  these  five  major  engineering 
sites  had  five  completely  different  process 
definitions  because  before  they  were  part 
of  a  common  business  unit,  they  were 
actually  competing  companies.  The 
Northeast  location  in  Marlborough, 
Massachusetts  where  I  work,  was  part  of 
the  original  Raytheon.  The  St.  Petersburg, 
Florida,  location  was  originally  part  of  E- 
Systems.  The  Fullerton,  California,  loca¬ 
tion  was  part  of  Hughes  Defense.  The 
Fort  Wayne,  Indiana,  location  was  also  part 
of  Hughes,  but  a  different  internal  organi¬ 
zation.  The  McKinney,  Texas,  location  was 
part  of  Texas  Instruments.  Each  had  their 
own  organizational  process  definition  and 
in  the  past  each  had  conducted  separate 


Capability  Maturity  Model  evaluations  and 
CMMl  appraisals  ranging  from  maturity 
levels  3  to  5  and  for  a  variety  of  discipline 
combinations:  software  or  software-sys¬ 
tems. 

Raytheon  has  defined  and  implement¬ 
ed  a  company-wide  Integrated  Product 
Development  System  (IPDS)  that  pro¬ 
vides  a  program-level  process  for  all 
Raytheon  programs.  IPDS  provides  an 
extensive  process  definition  describing 
what  should  be  done  throughout  a  pro¬ 
gram;  the  organizations  managing  pro¬ 
grams  had  to  define  processes  on  how 
IPDS  was  going  to  be  implemented  in 
their  programs.  These  organizational 
processes  were  the  ones  the  RNCS  orga¬ 
nizations  used  in  their  Standard  Capability 
Evaluation  (SCE)  or  SCAMPI s  and,  of 
course,  they  were  aU  different.  Then  the 


enlightened  senior  management  of  RNCS 
made  a  crucial  business  decision  that  all 
five  sites  were  going  to  have  a  common 
engineering  implementation  of  IPDS  with 
the  ultimate  goal  of  having  the  capability 
to  move  work  across  the  five  sites.  The 
five  sites  endeavored  over  a  couple  of 
years  to  define,  deploy,  and  implement  the 
RNCS  Common  Process  Architecture 
(CPA).  The  effort  ultimately  led  to  the  five 
major  engineering  centers  successfully 
conducting  our  SCAMPI  that  resulted  in  a 
Maturity  Level  5  rating  reported  to  the 
Software  Engineering  Institute  (SEI)  as 
Rjoytheon  Network  Centric  Systems  (Engineering 
—  sofitware  engineering,  sofitware,  hardware 
[SE/ SW/HW])  projects  with  engineering  devel¬ 
opment  in  the  scope  ofi  the  project  [2] . 

We  included  hardware  amplifications 
in  order  to  include  in  the  appraisal  our 
hardware  engineering  discipline.  In  addi¬ 
tion  to  —  and  key  to  —  our  success  was  our 
internal  program  engineering  discipline. 
The  key  to  our  successful  SCAMPI  was 
the  definition  and  deployment  of  the 
organizational  CPA  across  the  five  RNCS 
sites.  Let  us  examine  this  CPA  organiza¬ 
tional  process  definition. 

CPA 

The  design  of  the  CPA  followed  the  typ¬ 
ical  life  cycle  that  our  programs  follow. 
We  initially  started  creating  CPA  with  a 
requirements  definition  phase  where  we 
used  our  Raytheon  IPDS,  the  CMMl 
model,  and  International  Organization 
for  Standardization  (ISO)  9000,  to  pro¬ 
vide  the  requirements  for  the  CPA.  As  we 
defined  our  CPA,  the  identified  require¬ 
ments  were  entered  in  our  requirements 
database  and  then  were  flowed  down  to 
organizational  behavior  activities.  I  am 
not  going  into  the  details  of  the  overall 
CPA  process,  which  is  not  the  focus  of 
this  article.  However,  to  make  a  very  long 
story  short,  the  main  artifacts  of  the  CPA 
definition  were  a  series  of  CPA  Work 


Figure  1 :  Introduction  to  WP 

Work  Instruction 

Number 

U0141CRP 

Revision 
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Detailed  Planning 

Supersedes 

A 

Date 

01/22/07 

Introduction  The  purpose  of  the  Detailed  Planning  Wl  is  to  direct  a  program’s  detailed  engineering 

planning  activities  based  on  the  initial  planning  outputs  and  the  scope  of  the 
awarded  contract  as  part  of  preparing  for  the  program’s  Gate  5.  During  the  detailed 
planning  activities  those  work  products  from  the  initial  planning  activities  are  reviewed, 
and  updated  or  expanded  as  necessary.  Any  planning  work  products  that  may  have 
not  been  produced  as  a  result  of  the  initial  planning  activities,  but  are  determined  to  be 
required  after  contract  negotiations  will  need  to  be  created  as  part  of  the  detailed 
planning  activities. 

The  complete  set  of  detailed  planning  activities  is  defined  by  this  Wl  and  the 
supporting  Plan  Generation  Work  Instructions  for  the  various  program  plans. 

Refer  to  Appendix  A  for  an  overview  of  the  plans  hierarchy.  A  program  can  use  the 
NCS  Engineering  process  to  tailor  the  engineering  plans  identified  in  Appendix  A 
based  on  the  needs  of  the  program.  The  program  plans  should  be  put  under 
configuration  control  per  the  NCS  Work  Product  Management  Work  Instruction. 


Input(s) 


Contract  Documents 
Engineering  Cost  Estimates 
IPDS  Tailoring 
NCS  Engineering  Tailoring 
Program  Organization  Structure 
Initial  Engineering  Program  Plans 
Lessons  Learned  Repository 
Process  Assets  Library 
Work  Breakdown  Structure  (WBS) 
Engineering  Budget 
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Process  Summary  ID 

Steps  at  a  Glance 

Responsible  Role 

1 

Define  Planning  Strategy  for  Detailed  Planning 

Program  Engineer 
Program  Planning 

Team 

2 

Update  the  SEP,  HDP,  SDP,  and  QPP 

Program  Engineer 
Program  Planning 

Team 

3 

Update  Program’s  Life  Cycle  Model 

Program  Planning 

Team 

Note:  SEP  -  System  Engineering  Plan,  HDP  -  Hardware  Development  Plan,  SDP  -  Software  Development  Plan,  and  QPP  -  Quality  Program  Plan. 


Figure  2:  Detailed  Vlanning  Steps  at  a  Gland 


Instructions  (WI).  There  are  88  WIs  to  be 
exact.  Each  of  these  has  associated  with 
it  enablers  to  assist  in  process  implemen¬ 
tation,  such  as  the  following:  software 
tools,  templates  for  required  documents, 
process  checklists,  and  other  enablers  to 
be  used  by  aU  programs  at  the  five  engi¬ 
neering  sites.  Naturally,  the  CPA  organi¬ 
zational  process  definition  is  substantial 
in  detail  and  volume,  but  one  of  the 
major  cost  advantages  is  that  while  it  con¬ 
sists  of  a  large  amount  of  documenta¬ 
tion,  it  pales  in  size  to  having  five  differ¬ 
ent  sites  each  with  their  own  process  def¬ 
inition,  set  of  tools,  templates,  and 
enablers.  It  was  common  and  had  to  be 
used  by  all  programs.  The  CPA  has  to  be 
robust  enough  to  support  programs  with 
engineering  staffs  of  more  than  100  engi¬ 
neers  such  as  the  next  generation 
destroyer  for  the  Navy  or  the  replace¬ 
ment  of  all  the  Air  Traffic  Control  sys¬ 
tems  in  the  United  States.  Well,  CPA  is 
robust  enough  not  only  to  support  these 
substantial  programs  being  implemented 
across  the  five  engineering  sites,  but  also 
supports  CMMI  Maturity  Level  5 
processes. 

What  about  the  programs  that  do  not 
have  a  marching  army  of  engineers  with 
budgets  in  the  millions  and  billions  that 
are  executed  over  multiple  years?  How  do 
these  small  programs  deal  with  the  sub¬ 
stantial  organizational  process  that  was 
formally  appraised  at  CMMI  Level  5  and 
now  required  by  aU  RNCS  programs?  It 
can  be  wrapped  up  in  one  word:  tailoring. 
Tailoring  is  a  key  organizational  process 
that  is  used  by  large,  medium,  and  small 
programs.  All  programs  can  eliminate 
WIs  and  can  accept,  reject,  or  modify  the 
WI  requirements  as  well  as  modify  the 
contents  of  the  plans  as  needed  by  each 
program’s  contracts;  it  is  the  extent  and 
depth  of  the  tailoring  that  allows  the 
small  programs  to  survive.  However, 
before  getting  into  tailoring,  it  is  impor¬ 
tant  to  understand  the  general  structure 
of  the  CPA  WIs. 

CPA  WI  Structure  and 
Tailoring 

Every  WI  has  the  same  format  structure 
(see  Figures  1  through  4,  each  one  will  be 
detailed  in  this  section). 

Figure  1  is  the  first  part  of  the  CPA 
WI  that  provides  an  introduction  to  the 
WI  process,  WI  name  and  configuration 
management  information  of  the  WI.  It 
also  provides  the  input  necessary  for  the 
process.  This  particular  WI  is  the  main 
WI  used  by  the  typical  large  program  to 
perform  the  detailed  planning  of  the 


program  during  the  program  start-up 
phase  that  occurs  right  after  contract 
award.  The  Detailed  Planning  WI  defines 
all  the  steps  that  the  program  manage¬ 
ment  team  accomplishes  in  the  approxi¬ 
mately  45  days  after  the  contract  is 
signed.  In  actuality,  some  of  the  Detailed 
Planning  WI  steps  require  updates  to 
documents  and  other  artifacts  previously 
drafted  during  the  Initial  Planning  WI 
steps  that  were  accomplished  during  the 
proposal  stage  of  the  project.  From  the 
Initial  Planning  WI  there  are  several  out¬ 
puts  that  are  also  inputs  to  the  Detailed 
Planning  WI,  for  example:  Initial 
Engineering  Program  Plans  and  Work 
Breakdown  Structure  (WBS).  Typically, 
several  of  the  Initial  Engineering 
Program  Plans  (i.e..  Software 
Development  Plan  [SDP],  System 
Engineering  Plan  [SEP])  are  drafted  dur¬ 
ing  the  proposal  stage  and  in  the  begin¬ 
ning  of  the  program  after  contract  award 
during  program  start-up,  they  are 
reviewed  and  finalized.  Likewise,  the 


WBS  is  generated  during  the  proposal 
time  and  may  have  to  be  adjusted 
depending  on  the  results  of  contract 
negotiations.  A  portion  of  the  Detailed 
Planning  WI  is  shown  in  Figure  2. 

In  Figure  2,  you  see  the  first  three 
requirements  or  steps  at  a  glance  in  the  33 
steps  in  the  detailed  planning  process. 
Interestingly,  these  Steps  at  a  Glance 
were  flowed-down  as  requirements  from 
IPDS,  CMMI,  and  ISO9001  into  the 
detailed  planning  behavior  process 
(Figure  3). 

Figure  3  shows  the  expected  outputs 
from  when  the  CPA  Detailed  Planning 
WI  implemented  the  detailed  planning 
process,  which  included  multiple  engi¬ 
neering  plans  and  the  WBS.  These  are 
artifacts  produced  or  updated  as  the 
detailed  process  is  implemented.  Most  of 
the  plans  generated  from  this  work 
instruction  are  updates  to  those  drafted 
in  the  initial  planning  and  proposal 
phase.  Each  of  the  plans  is  enabled 
through  the  use  of  templates  with 


Figure  3:  Outputs  From  the  Detailed  Planning  Proeesd 


Output(s) 


Earned  Value  Management  System  (EVMS)  baseline 

Engineering  Cost  Estimates 

Engineering  Decision  Analysis  and  Resolution  Plan 

Engineering  Measurement,  Analysis  and  Improvement  Plan 

Engineering  Training  Plan 

Work  Project  Management  Plan  (WPMP) 

Gate  Review  Plan 

Hardware  Development  Plan 

Integrated  Master  Plan 

Integrated  Master  Schedule 

Integration,  Verification  and  Validation  Plan 

IPDS  Tailoring 

NCS  CPA  Tailoring 

Planning  Strategy 

Program  Organization  Structure 

Program’s  Security  Plan 

Quality  Program  Plan 

Risk  and  Opportunity  Management  Plan 

Software  Development  Plan 

Systems  Engineering  Plan 

WBS 
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Update 

Program’s  Life 
Cycle  Model 


Requirement 

Update  and  document  the  development  life-cycle  model  that  will  be  used. 

Guidance 

1 .  Review  and  update  as  necessary  the  program’s  life  cycle  phases  on  which 
to  scope  the  detailed  planning  effort.  Refer  to  the  Life  Cycle  Models  in  the 
Raytheon  Process  Assets  Library  (RayPAL),  for  descriptions  of  candidate 
models.  See  References  section  for  further  information  on  accessing  these 
models  in  RayPAL. 

2.  Document  the  life  cycle  phases  in  the  applicable  program  plans  such  as  the 
SEP,  SDP,  or  Hardware  Development  Plan  (HDP). 

3.  Update  the  systems  engineering  capabilities  expected  at  the  end  of  each 
applicable  life  cycle  phase  in  the  SEP. 

4.  Update  the  software  engineering  capabilities  expected  at  the  end  of  each 
applicable  life  cycle  phase  in  the  SDP. 

5.  Update  the  hardware  engineering  capabilities  expected  at  the  end  of  each 
applicable  life  cycle  phase  in  the  HDP. 


Figure  4:  Step  3  of  Detailed  Planning  Work  Instruction^ 


embedded  instructions  on  what  informa¬ 
tion  should  be  included  in  the  plans  (see 
Figure  4). 

Figure  4  shows  one  of  33  Steps  at  a 
Glance  in  the  Detailed  Planning  WL  It 
shows  how  each  step  is  expanded  into  the 
CPA  process  requirement  under  the 
Pequirement\i^2idivs\^  along  with  its  associat¬ 
ed  Guidance  on  how  to  implement  the 
Step  at  a  Glance  CPA  requirement. 
Naturally,  there  is  a  WI  for  tailoring  the 
WIs,  which  in  itself  can  also  be  tailored 
(see  Figure  5).  Along  with  the  Tailoring 
the  CPA  Process  WI,  the  organization  has 
provided  a  Guidance  enabler  on  tailoring 
that  includes  guidance  for  tailoring  small 
programs  as  well. 

Figure  5  shows  one  of  the  main  steps 
in  the  tailoring  process  -  this  is  actually 
step  8  of  12  steps  (see  Step  8  in  Figure  6). 

As  you  can  see,  most  of  the  WI 


describes  how  the  tailoring  process  is  con¬ 
ducted  (Steps  1-7  and  9-11  in  Figure  6), 
how  the  tailoring  decisions  are  stored  in 
iPlan  (Step  8),  and  how  the  tailoring  is 
approved  by  the  stakeholders  and  finally 
approved  by  the  ultimate  stakeholder 
(Step  12),  the  Engineering  Process  Group 
(EPG).  So,  a  program  cannot  go  wild  and 
tailor  out  important  portions  of  the  CPA. 
Tailoring  is  a  key  concept  in  the  CMMI 
model,  and  it  is  a  critical  task  for  programs 
using  our  CPA,  and  critical  to  survival  for 
small  programs  using  CPA.  The  model 
recognizes  that  many  factors  impact  the 
process  implementation  on  a  particular 
program,  for  example: 

•  Program  size. 

•  Program  complexity. 

•  Contractual  requirements. 

•  Customer  deliverable  requirements 
and  format  requirements. 


•  Specific  measurements  required  by  the 
organization  and  the  customer. 

As  far  as  our  CPA  tailoring,  each  of 
the  WI  requirements  can  be  accepted 
(i.e.,  accepted  as  written),  rejected  (i.e., 
deleted  from  the  work  instruction)  or 
modified  (i.e.,  rewritten,  such  as  SDP  will 
be  written  to  customer  format  versus 
CPA  format).  The  Guidance  for  each  of 
the  CPA  Requirements  can  be  adjusted 
for  the  specifics  of  the  program  and  con¬ 
tractual  requirements,  the  output  docu¬ 
ments  can  be  rejected  or  modified  to 
meet  the  contractual  requirements,  and 
they  can  be  adjusted  for  the  program  size. 
AU  tailoring  is  captured  in  the  iPlan  tool 
as  modified  WIs  for  the  specific  pro¬ 
gram.  The  iPlan  tool  provides  a  Program 
Tailoring  Workspace  —  a  virtual  library 
for  the  program’s  approved  process  defi¬ 
nition.  The  iPlan  tool  captures  all  the  tai¬ 
loring  done  to  each  of  the  CPA  WIs,  the 
EPG  approved  justification  for  all  the  tai¬ 
loring,  and  the  actual  modified  WIs  for 
reference  by  program  engineers  during 
implementation.  I  must  emphasize  again: 
that  aU  tailoring  is  approved  by  the  EPG 
before  the  program’s  tailored  process  is 
implemented.  For  larger  programs  with 
extended  schedules,  the  tailoring  of  the 
CPA  process  can  be  done  incrementally 
by  program  phase  so  you  do  not  have  to 
do  tailoring  of  the  system  test  process  if 
that  process  is  not  needed  for  several 
years.  This  process  flexibility  is  expected 
in  the  CMMI  and  is  an  integral  part  of 
the  RNCS  CPA  tailoring  process.  It  is 
especially  important  for  the  small  pro¬ 
grams  that  have  to  survive  among  the  big 
program  processes,  which  leads  us  to 
small  project  tailoring. 

Small  Project  Tailoring 

One  of  the  first  steps,  of  course,  is  to 
understand  what  it  means  to  be  a  small 
project.  In  the  beginning  of  CPA,  it  was 
more  a  concept  of  what  a  small  program 
consisted  of,  i.e.,  less  than  a  year  dura¬ 
tion,  small  budget,  small  team,  etc.  Small 
programs  are  now  formally  defined  as 
those  with  budgets  between  $500,000 
and  $4  million,  less  than  a  year  in  dura¬ 
tion,  and/or  a  team  of  eight  or  less  engi¬ 
neers  from  all  disciplines.  We  now  have  a 
definition  of  a  small  program  which  is 
helpful  when  we  enter  the  tailoring 
process.  The  tailoring  process  is  formal¬ 
ized  during  the  start  of  the  program. 
Now  one  of  the  major  structural  aspects 
of  Raytheon’s  IPDS  and  reinforced  in 
RNCS  CPA  is  the  program  gating 
process.  Formal  gates  are  reviewed  at 
specific  engineering  phases:  program 
start-up  (gate  5),  system  requirements 


Figure  5:  Main  Step  in  Tailoring  Work  Instructiord 

8.  Make  Tailoring  Requirement 

Decisions  Tailor  the  CPA  Work  Instruction  Requirements  (WIRs)  for  each  WI. 

Guidance 

1 .  For  each  work  instruction  provided  by  the  iPlan  tool,  the  tailoring  team 
evaluates  the  work  instruction  requirements  to  determine  the  program 
tailoring  decisions.  WI  Requirement  decisions  are  the  following: 

a.  Accepted 

b.  Rejected 

c.  Modified 

The  Tailoring  Decisions  and  Work  Instruction  tab  in  the  Tailoring  Workspace 
of  the  iPlan  tool  provides  visibility  into  the  WIRs  for  each  WI  and  allows  the 
program  to  select  or  deselect  specific  WIRs  and  provides  a  utilization 
summary  for  each  W I.  The  Library  Utilization  section  of  the  Work  Instruction 
tab  is  used  to  record  the  programs  tailoring  decisions  with  regard  to  a 
specific  set  of  WIRs.  Work  Instruction  Utilization  Acceptance  is  automatically 
determined  by  iPlan  based  on  the  following: 

a.  Accept  -  all  WIRs  accepted. 

b.  Modified  -  one  or  more  (but  not  all)  WIRs  rejected 

c.  Reject  -  all  WIRs  rejected. 

2.  This  process  is  repeated  for  all  work  instructions  provided  for  this  tailoring 
workspace.  Note  that  the  tailoring  workspace  may  be  different  for  each 
discipline  and  for  the  program  level  (multi -discipline). 

3.  The  tailoring  team  consults  with  the  EPG  representative  or  iPlan  SME  to 
evaluate  the  effect  on  higher  requirements  when  WIRs  have  not  been 
accepted.  Although  tailoring  is  not  to  be  considered  something  negative 
and  programs  should  do  what  is  smart  for  their  business,  the  organization 
has  an  interest  in  understanding  how  and  why  programs  are  deviating 
from  the  organizational  standards.  Keeping  with  this  notion,  when  a  program 
tailors  or  rejects  WIRs,  the  Notes  field  for  that  set  of  WIRs  must  be  filled  in 
to  address  ‘why’  the  program  is  deviating  from  the  organizational  standard. 
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review  (gate  6),  preliminary  design  (gate 
7),  detailed  design  (gate  8),  formal  testing 
(gate  9),  etc.  As  a  matter  of  fact,  we  have 
11  formal  gates,  four  before  contract 
award  and  seven  gates  after  contract 
award.  The  important  one  for  small  pro¬ 
ject  tailoring  is  our  gate  5,  known  as  pro¬ 
gram  start-up.  Every  program  right  after 
contract  award  has  45  days  to  complete 
the  program  start-up  process  and  review 
the  overall  program  approach  with  the 
various  senior  management  levels  associ¬ 
ated  with  the  execution  of  the  program. 
Key  to  this  start-up  process  is  the  tailor¬ 
ing  of  the  CPA  WI  to  describe  how  each 
WI  is  going  to  be  implemented  on  the 
program.  As  previously  stated,  in  the  pro¬ 
posal  phase,  several  of  the  engineering 
plans  may  have  been  written  in  draft 
form  or  written  as  part  of  the  proposal. 
After  contract  award,  during  the  program 
start-up  phase,  the  various  engineering 
plans  are  either  going  to  be  updated  from 
the  drafts  written  in  the  proposal  phase 
or  written  in  the  start-up  phase.  The  engi¬ 
neering  plans  consist  of  the  following: 

•  SEP. 

•  SDP 

•  HDP 

•  Stakeholder  Involvement  Plan  (SIP). 

•  Measurement  Analysis  and  Improve¬ 
ment  Plan  (MAIP). 

•  Integration,  Verification  and  Vali¬ 
dation  Plan  (IWP). 

•  Quality  Program  Plan  (QPP). 

•  WPMP  and  Work  Product  List  (WPL) . 

•  Several  others. 

As  you  can  imagine,  this  can  be  an 
intensive  period  for  the  managers  involved 
with  the  program.  The  small  program  tai¬ 
loring  of  the  CPA  WIs  is  a  heavy  duty 
effort.  Just  think  of  trying  to  pare  down 
88  WIs  in  order  to  describe  how  you  are 
going  to  implement  the  program  using 
these  tailored  CPA  WIs.  The  first  group  of 
small  programs  that  pioneered  their  way 
through  this  tailoring  process  and  tailoring 
justifications  were  reused  as  the  various 
plans  were  adjusted  and  reused.  Well,  it 
did  not  take  long  to  determine  that  an 
organizational  process  improvement  was 
necessary  to  assist  these  small  programs 
through  the  program  start-up  and  tailor¬ 
ing  processes:  enter  the  Small  Program 
Variant  (SPV)  Planning  WI. 

SPY  Planning 

The  RNCS  CPA  Engineering  Councils 
collaborated  and  generated  a  new  WI  to 
be  used  by  small  programs  to  do  their 
program  planning  called  the  SPV 
Planning  WI.  It  essentially  provided  pre¬ 
tailoring  of  the  CPA  WIs  as  a  starting 
point.  One  of  the  major  steps  in  the  SPV 


planning  was  the  creation  of  the  Small 
Program  Engineering  Plan  (SPEP)  tem¬ 
plate.  The  SPEP  template  combined  the 
major  portions  of  the  SEP,  SDP,  HDP, 
and  IWP  into  the  single  SPEP. 
Furthermore,  additional  planning  activi¬ 
ties  associated  with  large  programs  were 
reduced  for  the  small  programs.  For 
example,  the  amount  of  measurement 
data  is  reduced,  such  as  staffing,  since  it  is 
easy  to  keep  track  of  three  or  four  people. 
Typically,  small  programs  deal  with  only 
one  or  maybe  two  of  the  engineering  dis¬ 
ciplines,  so  specific  discipline  WIs  can  be 
eliminated,  for  example:  A  software-only 

*%ey  to  this  start-up 
process  is  the  tailoring 
of  the  CPA  WI  to 
describe  how  each  WI  is 
going  to  be  implemented 
on  the  program  ...  the 
engineering  plans  may 
have  been  written  in 
draft  form  or  written  as 
part  of  the  proposal.*^ 

job  can  eliminate  all  the  WIs  associated 
with  Hardware  Preliminary  Design, 
Hardware  Detailed  Design,  Hardware 
Testing,  etc.  The  details  of  stakeholder 
involvement  can  be  reduced  on  small 
teams.  Details  such  as  formal  weekly  team 
meetings  can  be  eliminated  especially 
when  the  team  is  two  to  three  people  all 
working  in  the  same  cubicle.  Formal 
reviews  are  reduced  to  three  to  four  page 
review  packages  versus  35  to  40  page 
detailed,  senior  manager  review  packages. 


Small  programs  still  have  to  deal  with  78 
WIs  and  the  tailoring  of  those,  but  with 
the  creation  of  the  SPEP  a  significant 
work  load  was  removed  from  the  pro¬ 
grams.  There  are,  however,  in  the  SPV 
Planning  WI  some  of  the  other  engineer¬ 
ing  plans,  even  for  the  small  program, 
that  have  to  be  created  and  implemented 
such  as  the  following:  the  Risk  and 
Opportunity  Management  Plan,  the 
MAIP,  the  WPMP,  WPL,  and  the  SIP.  On 
my  small  programs,  I  have  even  been  able 
to  convince  the  EPG  that  some  of  these 
required  standalones  could  be  down- 
scoped  and  incorporated  into  my  pro¬ 
gram’s  single  SPEP,  thus  reducing  the 
coordination  and  sign-off  cycle  of  multi¬ 
ple,  individual  plans;  it  was  done  using 
just  my  one  SPEP. 

Other  allowable  tailoring  is  the  com¬ 
bination  of  engineering  phases.  Typically, 
small  programs  are  associated  with 
extending  the  functionality  of  product 
lines  where  the  preliminary  design  of  the 
functionality  is  already  known.  So,  small 
programs  can  tailor  the  conduct  of  two 
of  the  engineering  gates  reviews  into  one: 
the  preliminary  design  and  detailed 
design  phases  and  the  associated  gates  7 
and  8  (Preliminary  Design  Review  [PDR] 
and  Critical  Design  Review  [CDR]).  Even 
more  typical,  the  system  requirements  are 
usually  small  changes  to  the  product  line 
and  the  System  Functional  Requirements 
Review  (gate  6)  can  also  be  rolled  into  the 
combined  PDR/ CDR.  This  is  all  depen¬ 
dent  upon  the  tailoring  and  permission 
given  by  the  EPG  during  the  EPG’s  tai¬ 
loring  approval  meeting. 

Implementation  of  Small 
Programs 

With  the  SPV  Planning  WI,  how  does  the 
implementation  of  small  programs  differ 
from  the  standard  programs?  To  start,  the 
number  of  WIs  to  tailor  is  reduced  to  78 
WI  for  SPV  down  from  the  88  used  in 
standard  programs.  The  combined  SPEP 
document  combines  several  of  the 


Figure  6:  Sfeps  at  a  Glance  of  the  Tailoring  ProcesT 

ID  Steps  at  a  Glance 

1  Establish  Tailoring  Team 

2  Use  CPA  Tailoring  Guidelines 

3  Establish  Tailoring  Workspace 

4  Execute  T ailoring  Kickoff 

5  Select  Program  Characteristics 

6  Identify  Stakehoiders 

7  Document  Taiioring  Stakehoiders 

8  Make  Taiioring  Decisions 

9  Document  Pianning  Strategy 

10  Capture  Tailoring  Workspace 

1 1  Schedule  Stakeholder  Review 

12  Obtain  Stakeholder  Approval 


Responsible  Role 

Program  Engineer 
Tailoring  Team 
Tailoring  SME 
Program  Engineer 
Tailoring  Team 
Tailoring  Team 
Tailoring  Team 
Taiioring  Team 
Taiioring  Team 
Program  Engineer 
Program  Engineer 
Program  Engineer,  Taiioring 
Stakehoiders 
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required  plans  and  averages  around  45-60 
pages.  The  volume  of  the  individual  plans 
needed  for  standard  programs  collectively 
can  be  in  the  hundreds  of  pages. 
Measurements  are  key  concepts  in  high 
maturity  organizations.  Regardless  of  the 
size  of  the  program  (standard,  small,  or 
micro)  they  need  to  collect  the  typical 
EVMS  metrics  needed  to  analyze  the  fol¬ 
lowing:  Schedule  Performance  Index  (SPI) 
and  Cost  Performance  Index  (CPI).  The 
difference,  however,  is  the  number  of  cost 
accounts  that  are  tracked  by  small  pro¬ 
grams  are  around  30  to  40,  where  in  stan¬ 
dard  programs  the  number  of  cost 
accounts  can  be  in  the  hundreds  of 
account  numbers.  Naturally,  on  small  pro¬ 
grams  the  number  of  data  points  collect¬ 
ed  is  quite  limited.  Neverless,  there  are 
other  measurements  recommended  in  the 
small  program  tailoring  guidelines  in  addi¬ 
tion  to  CPI/SPI  to  manage  these  small 
programs.  From  the  Tailoring  Guidelines: 

The  metrics  that  a  [small]  program 
collects  is  based  on  the  characteris¬ 
tics  of  the  program  and  customer 
requirements.  The  following  is  a 
recommended  list  of  metrics  that  a 
small  program  should  collect 
unless  there  is  a  valid  reason  to  tai¬ 
lor  out,  such  as: 

•  CPI/SPI. 

•  Defect  Containment  [which  are 
collected  through  peer  review/ 
inspection  data  and  system  trouble 
reporting  data]. 

•  Requirements  volatility. 

•  Any  (i.e.  hardware,  software,  sys¬ 
tems)  applicable  engineering  pro¬ 
ductivity  measures.^ 

The  small  program  task  managers  indi¬ 
cate  what  measurements  they  use  to  man¬ 
age  their  programs  in  their  MAIP.  The 
MAIP  which  identifies  the  program’s  mea¬ 
surements  requires  approval  by  the  EPG. 
Other  adjustments  are  allowed  for  small 
programs  during  the  tailoring  process, 
such  as  the  following: 

•  Even  though  they  started  with  78 
potential  WIs,  and  since  the  disciplines 
such  as  hardware  can  be  tailored  out  if 
there  is  no  hardware  development,  the 
number  of  applicable  WIs  can  get 
down  to  about  20  and  those  can  be 
substantially  tailored  even  further. 

•  Monthly  review  packages  (35-40 
slides)  are  reduced  to  four  square 
charts  (three  to  four  slides). 

•  Process  Support  Team  (PST)  meetings 
where  metrics  analysis  and  process 
compliance  checks  are  reduced  both  in 
the  team  size  and  the  time  to  conduct 


the  PST  meetings. 

•  The  10  required  plans  can  be  com¬ 
bined  with  the  SPEP,  the  implementa¬ 
tion  of  the  various  discipline  plans  are 
not  as  detailed,  and  the  associated 
enablers  and  templates  are  also 
reduced  in  size. 

All  the  tailoring,  even  though  much 
reduced,  still  requires  a  significant  effort 
by  the  small  program’s  management  team. 
Therefore,  the  next  step  is  for  the  EPGs 
and  the  senior  level  discipline  councils  to 
provide  even  more  pre-approved  tailoring 
for  each  of  the  WIs  and  the  WIs  associat¬ 
ed  enablers.  Going  one  step  further,  the 
foUow-on  pre-tailoring  activity  wiU  be  to 
address  micro-programs.  These  micro¬ 
programs  are  even  smaller  than  the  small 
programs  I  just  discussed.  The  micro-pro- 
grams  typically  have  work  effort  content 
between  1,000  and  8,000  staff  hours. 
Currently,  there  are  a  number  of  WIs 
being  piloted  by  the  RNCS  organization 
that  target  these  micro-programs.  Yes,  I 
know  what  you  are  thinking  —  what  about 
the  projects  under  1,000  hours  ...  small 
projects  among  big  trees;  ever  think  about 
where  toothpicks  come  from? 

Summary 

If  we  used  the  CMMI  through  Maturity 
Level  5  itself  as  a  process  description,  it 
would  be  well  over  600  pages.  So  when 
you  establish  your  standard  processes 
and  actually  write  the  organizational 
process  description  you  can  see  that  all 
the  process-oriented  documentation  and 
implementation  of  these  processes  can 
become  an  insurmountable  issue  for 
small  programs  (e.g.,  small  budget,  small 
team,  short  schedule).  The  CMMI  rec¬ 
ognizes  that  tailoring  is  key  to  the  suc¬ 
cessful  implementation  of  the  organiza¬ 
tional  processes  for  the  variety  of  pro¬ 
grams  that  organizations  have  in  their 
business.  It  is  up  to  the  organization  to 
provide  the  guidance,  methodology,  and 
the  expectations  of  the  small  programs 
that  are  being  executed  in  the  same 
process  environment  as  the  big  trees.  An 
organization  has  to  have  an  organiza¬ 
tional-approved  tailoring  method  for  the 
small  programs  in  order  to  survive  and 
be  successful,  while  still  supporting  the 
essence  of  the  organizational  processes. 
Tailoring  is  a  life-saver  for  our  small  pro¬ 
grams  that  also  support  our  overall 
CMMI  Maturity  Level  5.^ 
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Field  Guide  to  Provide  Step-by-Step  Examples  for  Improving 

Processes  in  Small  Settings 

Caroline  Graettinger,  Suzanne  Garcia,  Christian  Carmody  M.  Lynn  Penn 

and  William  Peterson  University  of  Pittshurgb  Medical  Center  Uockheed  Martin  Corporation 

Software  Engineering  Institute 

To  help  organisations  in  small  settings  pursue  process  improvement,  the  Software  Engineering  Institute  (SEI)  is  inviting  con¬ 
tributors  to  help  develop  a  field  guide  with  how-to  guidance,  examples,  templates,  checklists,  and  other  information. 


If  you  work  for  a  small  business,  partici¬ 
pate  on  a  small  team,  belong  to  a  small 
business  unit,  or  work  on  small  projects, 
then  you  work  in  a  small  setting  and  are  like¬ 
ly  familiar  with  the  challenges  of  process 
improvement  in  these  contexts. 

Consider  the  following  example  from  a 
small  business:  The  Chief  Executive  Officer 
complains  that  she  does  not  have  enough 
cycles  or  resources  to  try  out  even  one  of 
several  process  improvement  concepts  that 
her  various  customers  are  asking  for.  She 
recognizes  the  risks  of  inaction,  but  the  cost 
seems  prohibitive,  and  she  has  never  used 
consultants  before.  Her  employees  are  all 
busy  and  many  believe  that  ifs  always  worked 
fine  the  way  it  is.  She  would  like  to  have  repeat- 
able,  predictable  work  processes  that  pro¬ 
duce  quality  products  and  services  and 
enable  her  company  to  stand  out  from  the 
crowd.  She  knows  she  needs  some  kind  of 
process  improvement  activity.  But  is  her 
company  ready?  Is  she  ready?  How  can  they 
become  ready?  What  can  they  do  them¬ 
selves,  and  how  can  they  be  better  con¬ 
sumers  of  process  improvement  products 
and  services? 

The  SEI  has  heard  stories  like  this  from 
small  settings  around  the  world,  and  began 
to  explore  this  arena  in  2003  and  2004  with 
a  pilot  study  in  Huntsville,  Alabama.  The 
study  resulted  in  new  knowledge  and  ideas 
for  how  to  accelerate  implementation  of 
one  improvement  methodology  -  Capability 
Maturity  Model  Integration  (CMMI®)  —  in 
small  businesses  [1,2].  This  was  followed  by 
an  insightful  workshop  in  October  2005 
involving  researchers  from  around  the 
world  [3]. 

Based  on  this  work  and  recommenda¬ 
tions  from  the  International  Process 
Research  Consortium,  the  SEI  launched  the 
Improving  Processes  in  Small  Settings 
(IPSS)  project,  in  collaboration  with  Uni¬ 
versity  of  Pittsburgh  Medical  Center 
(UPMC)  and  Lockheed  Martin  Corporation 
(LMCO).  Why  would  UPMC  and  LMCO  - 
whose  employees  number  in  the  tens  or 
hundreds  of  thousands  —  be  interested  in  a 
project  for  small  settings?  Because,  like 
many  large  organizations,  they  are  amalgams 


of  many  small  projects  and  business  units, 
with  myriad  small  business  partners  and 
suppliers. 

The  first  IPSS  project  is  the  Field  Guide 
for  Improving  Processes  in  Small  Settings. 
The  guide  is  not  constructed  like  CMMI  or 
any  other  process  improvement  models  or 
frameworks;  it  is  a  collection  of  how-to 
guidance  for  process  improvement  in  small 
settings,  independent  of  the  process  model 
or  standard  used.  We  intend  it  to  help  fast- 
track  the  improvement  effort  and  convey 
the  scope  of  effort  and  skills  involved  at 
each  step  so  that  the  small-setting  practi¬ 
tioner  can  be  a  smarter  consumer  of 
process  improvement  products  and  services 
or  be  better  at  doing  it  themselves,  whichev¬ 
er  they  choose. 

The  information  in  the  field  guide  is 
organized  under  six  competencies:  (1)  build¬ 
ing  and  sustaining  sponsorship  and  owner¬ 
ship;  (2)  developing  and  measuring  realistic 
goals;  (3)  developing  and  sustaining  a 
process  improvement  infrastructure;  (4) 
defining  and  describing  processes;  (5)  devel¬ 
oping  new  or  improved  processes;  and  (6) 
determining  improvement  progress.  Each 
competency  comprises  a  set  of  activities 
that  describe  what  to  do  to  achieve  that  com¬ 
petency,  and  each  activity  comprises  a  set  of 
tasks  that  explain  how  to  do  each  activity. 

Our  plan  for  populating  the  field  guide 
includes  collecting  real-world  experiences 
from  experts  across  the  process  community 
who  can  provide  knowledge,  techniques, 
examples,  checklists,  scripts,  and  other  arti¬ 
facts  to  help  others  succeed  in  small  settings. 
The  guidance  will  include  step-by-step  tasks 
for  various  situations  and  constraints  of  the 
small  setting. 

We  welcome  the  involvement  of  small 
settings  experts,  citizens,  and  their  stake¬ 
holders  to  help  accelerate  the  development 
of  the  field  guide.  There  are  currendy  three 
ways  to  participate: 

•  Become  a  project  affiliate  and  work 
directly  on  the  guide  at  various  stages. 

•  Become  a  project  sponsor,  which 
enables  organizations  to  influence  the 
priority  order  of  the  guide’s  development 
and  content  for  their  particular  needs. 


•  Complete  the  brief  survey  we  have  cre¬ 
ated  to  collect  information  at  <www.sei. 
cmu.edu/iprc/ipss.html>. 

Eor  more  information  on  the  field  guide 

and  the  IPSS  project,  please  visit  <www.sei. 

cmu.edu/iprc/ipss.html>.^ 
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Proceedings  of  the  First 
International  Research 
Workshop  for  Process 
Improvement  in  Small 
Senings,  2005 

www.sei.cmu.edu/publications/ 
documents/ 06. reports/ 06sr00 1  .html 
The  first  International  Research 
Workshop  for  Process  Improvement  in 
Small  Settings  was  held  October  19-20, 
2005  at  the  Software  Engineering 
Institute  in  Pittsburgh,  Pennsylvania. 
Attendees  from  Australia,  Canada,  Chile, 
China,  Germany,  Ireland,  India,  Japan, 
Malaysia,  Mexico,  Spain,  and  the  United 
States  discussed  the  challenges  of  process 
improvement  in  small  and  medium  size 
enterprises,  small  organizations  within 
large  companies,  and  small  projects.  The 
presentations  addressed  starting  and  sus¬ 
taining  process  improvement,  qualitative 
and  quantitative  studies,  and  using 
Capability  Maturity  Model  Integration 
(CMMI),  Agile,  Modelo  de  Procesos 
para  la  Industria  de  Software, 
International  Organization  for  Stan¬ 
dardization,  Quality  Function  Deploy¬ 
ment,  and  Team  Software  Process  in 
small  settings.  The  workshop  also  had 
working  groups  that  discussed  issues 
unique  to  small  settings,  such  as  regional 
support  centers  and  process  improve¬ 
ment  on  a  shoestring.  This  report 
includes  the  papers  from  this  workshop 
and  presents  conclusions  and  next  steps 
for  process  improvement  in  small  set¬ 
tings.  This  report  also  contains  the  work¬ 
shop  breakout  session  results. 

Extreme  Programming 

WWW,  extremeprogramming,  org 
Extreme  Programming  (XP)  is  a  deliber¬ 
ate  and  disciplined  approach  to  software 
development.  About  eight  years  old,  it 
has  already  been  proven  at  many  compa¬ 
nies  of  all  different  sizes  and  industries 
world  wide.  This  web  site  gives  an  over¬ 
all  view  of  XP  and  presents  numerous 
resources  for  learning  more  about  the 
programming  method. 

Software  Technology 
Support  Center 

www.stsc.hill.af  mil 

In  1987,  the  U.S.  Air  Force  selected 
Ogden  Air  Logistics  Center  (OO-ALC), 
Hill  Air  Force  Base,  Utah,  to  establish 
and  operate  its  Software  Technology 
Support  Center  (STSC).  It  was  chartered 


to  be  the  command  focus  for  proactive 
application  of  software  technology  in 
weapon,  command  and  control,  intelli¬ 
gence  and  mission-critical  systems.  The 
STSC  provides  hands-on  assistance  in 
adopting  effective  technologies  for  soft¬ 
ware-intensive  systems.  We  help  organi¬ 
zations  identify,  evaluate,  and  adopt 
technologies  that  improve  software  prod¬ 
uct  quality,  production  efficiency  and 
predictability.  We  help  others  buy  and 
build  software  and  systems  better.  We 
use  the  term  technology  in  its  broadest 
sense  to  include  processes,  methods, 
techniques,  and  tools  that  enhance 
human  capability.  Our  focus  is  on  field- 
proven  technologies  that  will  benefit  the 
Department  of  Defense  mission. 

Software  Engineering 
Institute’s  Capability 
Maturity  Model  Integration 
Web  Site 

www.sei.cmu.edu/cmmi 
The  CMMI  is  a  process  improvement 
approach  that  provides  organizations 
with  the  essential  elements  of  effective 
processes.  It  can  be  used  to  guide  process 
improvement  across  a  project,  a  division, 
or  an  entire  organization.  CMMI  helps 
integrate  traditionally  separate  organiza¬ 
tional  functions,  set  process  improve¬ 
ment  goals  and  priorities,  provide  guid¬ 
ance  for  quality  processes,  and  provide  a 
point  of  reference  for  appraising  current 
processes.  This  page  points  you  to  places 
where  you  can  find  more  information 
about  CMMI,  and  describes  the  world¬ 
wide  adoption  and  benefits  of  CMMI. 

Practical  Software  and 
Systems  Measurement 

www.psmsc.com 

Practical  Software  and  Systems 
Measurement  (PSM)  was  developed  to 
meet  today's  software  and  system  techni¬ 
cal  and  management  challenges.  It  is  an 
information-driven  measurement  process 
that  addresses  the  unique  technical  and 
business  goals  of  an  organization.  The 
guidance  in  PSM  represents  the  best 
practices  used  by  measurement  profes¬ 
sionals  within  the  software  and  system 
acquisition  and  engineering  communi¬ 
ties. 
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29  April  -  2  May  2008  •  LAS  VEGAS,  NEVADA 

Technology:  Tipping  the  Balance 


Welcome  to  the  20th  installment  of  the  Systems  and 
Software  Technology  Conference  (SSTC).  Just  about 
20  years  ago.  Tetris  was  becoming  the  computer  game 
of  choice.  VGA  Graphics  were  gaining  in  popularity.  A 
company  known  by  its  initials,  IBM,  was  delivering  the 
PS/2  computer.  It  had  a  funny  new  pointing  device  - 
mouse.  And  the  first  ever  Software  TechnologyiGonference 
was  held.  From  the  beginning,  SSTC  has.,ftjt:used  on 
technology  relating  to  the  Departme)jtof  Defense.  We 
begin  another  decade  with  this  spme  focus. 

Our  theme  will  be  "Techtu^ogy:  Tipping  the  Balance". 
The  idea  behind  the  theme  is  to  explore  new  and  needed 
technologies,  as  well  as  lessons  learned,  which  tip  the 
balance  in  the  favor  of  our  Defense  Services,  providing 
them  an  asymmetric  advantage. 


Submittal  Topics  Included: 

•  Assurance  and  Security 

•  Estimating  and  Measuring 

•  Lessons  Learned 

•  ftew  Concepts  and  Trends 

•  Policy  and  Standards 

•  Processes  and  Methods 

•  Professional  Development 

•  Robust  Engineering 

•  Systems:  from  Design  to  Delivery 


Who  Should  Attend: 

•  Acquisition  Professionals 

•  Program/Project  Managers 

•  Programmers 

•  System  Developers 

•  Systems  Engineers 

•  Process  Engineers 

•  Quality  and  Test  Engineers 


For  Complete  Conference  &  Trade  Shaw  Information 

www.sstc-on  I  ine.org 

REGISTER  TODAY! 
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Crosstalk’s  20*”  Anniversary 


Crosstalk,  The  Journal  of  Defense  Software 
Engineering,  is  celebrating  its  20^''  anniversary. 

From  its  inception.  Crosstalk’s  goal  has  been 
to  inform  and  educate  readers  on  software 
engineering  processes,  policies,  and  other 
technologies.  We  love  hearing  from  our  readers  and 
friends  on  how  we  are  doing.  As  a  free  journal, 
these  comments  are  the  lifeblood  of  our  existence: 
They  keep  our  staff  focused  and  our  sponsors 
motivated  and  pleased.  If  you  find  reading 
Crosstalk  saves  you  time  and  money,  helps 
improve  your  processes,  has  helped  save  your 
project,  or  makes  your  life  easier,  we  want  to  know 
about  it.  The  comments  we  receive  will  be 
considered  for  inclusion  in  our  anniversary  issue 
this  coming  August. 

Please  send  your  feedback  to  Crosstalk’s 
Publisher,  Beth  Starrett  at  beth.starrett@hill.af.mil. 
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Small  Boats  Among  the  Big  Ships 


You’ve  probably  heard  of  the  U.S. 

Navy’s  famous  Patrol  Torpedo  (PT) 
boats  of  World  War  IL  Originally 
armed  with  four  machine  guns  and  four 
torpedoes,  they  were  pound-for-pound 
the  Navy’s  most  heavily  armed  vessels. 
The  80-foot  wooden  warships  served 
throughout  the  Pacific,  in  the 
Mediterranean,  and  even  in  the  English 
Channel.  Their  most  famous  actions 
include  evacuating  General  Douglas 
Mac  Arthur  from  the  Philippines  and 
assisting  John  F.  Kennedy’s  exploits 
when  his  PT109  was  cut  in  half  by  a 
Japanese  destroyer.  By  the  end  of 
World  War  II,  the  PT  boats  had  racked 
up  an  impressive  list  of  accomplish¬ 
ments  [1]. 

Why  bring  up  PT  boats  in 
Crosstalk?  Sure,  the  editors  like 
stories  framed  with  little-known  bits  of 
military  history,  but  they’re  an  excellent 
analogy  for  this  issue’s  Small  Projects,  Big 
Issues  theme. 

At  the  war’s  inception,  the  PT  boat 
retained  the  basic  mission  of  all  battle¬ 
ship-age  torpedo  boats:  Use  stealth  and 
speed  to  sink  a  capital  ship  using  torpe¬ 
does.  Initially,  the  enclosed  waters  of 
the  Philippine  and  Solomon  Islands 
provided  opportunities  to  continue 
their  historical  mission.  But  opportuni¬ 
ties  for  surface  combat  would  soon 
wane.  Aircraft  and  radar  made  it  much 
more  difficult  for  the  PTs  to  get  close 
enough  to  use  their  powerful  stings. 

Despite  this,  the  PTs  found  them¬ 
selves  even  more  in  demand.  Scouting, 
interdicting  enemy  barge  traffic,  per¬ 
forming  reconnaissance,  rescuing 
downed  flyers,  giving  close  shore  con¬ 
voy  support,  gathering  intelligence, 
supporting  ground  operations,  and 
many  other  missions  that  came  the  PTs’ 
way.  They  soon  bristled  with  new 
weapons:  auto-cannon,  mortars,  and 
rockets.  Some  PTs  abandoned  their  tor¬ 
pedoes  entirely  for  more  guns! 
Occasionally,  though,  they  were  still 
able  to  slam  a  torpedo  into  a  major  war¬ 
ship,  as  they  did  during  the  Battle  of 
Leyte  Gulf  —  the  largest  naval  battle  in 
modern  history  [2]. 

Small  projects  can  learn  important 
lessons  from  the  small  boats.  First,  the 
PTs  were  able  to  change  their  basic  mis¬ 
sion  of  hunting  capital  surface  ships  by 


adapting  to  fill  an  important  niche  in 
overall  naval  strategy.  Originally 
designed  for  a  role  in  the  big  ship  navy, 
the  PT  boats  found  that  circumstances 
dictated  a  very  different  one.  They 
adapted  well  to  their  new  role  due  in 
large  part  to  the  compactness  of  the 
boat  and  its  crew.  All  teams  must  be 
able  to  adapt,  but  change  comes  more 
easily  to  smaller  entities.  This  built-in 
flexibility  allows  small  teams  to  operate 
with  less  stringent  operating  processes, 
often  much  less. 

More  importantly,  consider  how  a 
PT  boat  differed  from  other  U.S.  Navy 
warships.  Table  1  compares  a  PT  boat 
to  a  heavy  cruiser.  Both  are  warships, 
but  the  differences  are  striking.  Crew 
training  cannot  be  over-emphasized. 
Losing  even  one  man  on  a  small  crew 
could  be  devastating  if  another  sailor 
couldn’t  take  over.  So  every  PT  sailor 
had  to  be  able  to  do  almost  any  job  on 
the  boat,  just  like  a  small  project  soft¬ 
ware  developer  needs  to  be  able  to  do 
any  job  on  a  project:  requirements, 
design,  coding,  information  and  tech¬ 
nology,  documentation,  configuration 
management,  and  even  leadership. 

For  leadership,  consider  the  PT  boat 
skipper.  He  knows  each  sailor  in  his 
crew  personally.  His  chain  of  com¬ 
mand:  one  executive  officer.  Before  a 
mission,  he  can  muster  the  entire  crew 
for  a  direct,  verbal  briefing.  He  knows 
the  complete  capabilities  of  the  boat: 
weapons,  engines,  communications,  and 
performance.  From  the  cockpit,  he  can 
issue  orders  to  any  member  of  the  crew 
verbally  or  via  hand  signal.  He’s  confi¬ 
dent  his  cross-trained  crew  can  step  in 
should  a  shipmate  be  disabled.  The 
short,  focused  PT  boat  missions  allevi¬ 
ate  him  from  the  more  mundane 


aspects  of  captaincy.  In  battle,  he  oper¬ 
ates  alone  or  with  other  PT  boats,  all 
with  the  same  modus  operandi.  Naval  doc¬ 
trine  of  the  day  was  developed  for  the 
larger  ships,  but  the  realities  of  the  PT’s 
missions  made  it  clear  that  they  needed 
different  methods,  different  tactics  — 
different  processes  —  to  achieve  that  result. 
Aided  by  their  compact  nature,  the  PT 
crews  readily  established  their  own  suc¬ 
cessful  processes. 

If  your  project  calls  for  a  PT  boat, 
put  one  in  place  and  run  it  accordingly. 
Using  processes  and  procedures  estab¬ 
lished  for  the  normal  behavior  expected 
of  large-scale  projects  can  easily  be 
counterproductive  —  or  worse.  Don’t 
operate  a  PT  boat  like  a  heavy  cruiser 
or  expect  it  to  act  like  one.  If  you  do, 
you’ll  be  sunk. 

— Dan  Knauer 
TLA 

A  Three-Letter  Acronym  Corporation 
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Additional  Reading 

The  Internet  provides  several  excellent 
sources  of  information  on  the  PT  boats 
including: 

1.  The  Historical  Naval  Ships  Organi¬ 

zation  <www.hnsa.org>. 

2.  PT  Boats  Inc  <www.ptboats.org>. 

3.  John  Drain’s  PT  Boat  Site  <www. 
pt-boat.com>. 


Table  1 :  PT  Boat  Versus  Navy  Warship 


Aspect 

PT  Boat 

Heavy  Cruiser 

Composition 

Wood 

Steel 

Engines 

Three  Packard  V-12  motors 

Boiler-driven  turbines 

Fuel 

Aviation  gasoline 

Oil 

Armament  ratio 

One  weapon  per  man 

One  weapon  per  20  men 

Transport  to  operating  area 

Carried  aboard  ship 

Arrived  under  own  power 

Mission  duration 

Nightly  patrols  returning  to 
same  base 

Multi-week  cruises 
returning  to  varying  ports 

Crew  training 

Cross-trained  in  two 
disciplines;  familiar  with  all 
tasks  on  boat 

Single  assignment  plus 
battle  station 
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